Users' Guide on Measuring Fragility

8/14/2019 Users' Guide on Measuring Fragility

1/152


2/152


3/152

U Gu Measuring Fragility

Authors

Javier Fabra Mata, UNDP

Sebastian Ziaja, DIE

Editors

Jrg Faust, DIE

Joachim Nahem, UNDP

Geman develpmen ine / dece in Enwcklngplk (diE)une Nan develpmen Pgamme (uNdP)


4/152

uNdP dclame:The views expressed in this publication are the authors and do not necessarily represent those o the United

Nations, including UNDP, or its Member States.

F e nman pleae cnac:

Geman develpmen ine/ une Nan develpmen Pgammedece in Enwcklngplk Bureau or Development Policy

Tulpeneld 6 Democratic Governance Group

53113 Bonn, Germany Oslo Governance Centre

Inkognitogata 37, 0256 Oslo, Norway

Tel: +49 (0)228 94927-0 Tel: +47 23 06 08 20

Fax: +49 (0)228 94927-130 Fax: +47 23 06 08 21

www.die-gdi.de www.undp.org/oslocentre

Copyright 2009 by the German Development Institute / Deutsches Institut r Entwicklungspolitik (DIE) and the United Nation

Development Programme (UNDP). All rights reserved.

For any errors or omissions ound subsequent to printing, please visit our websites.


5/152

L abbevan v

Acknwlegemen vFew by e uNdP ol Gvenance Cene v

Few by e Geman develpmen ine / dece in Enwcklngplk v

iNtroduCtioN: ABout this GuidE 1

PArt i: AssEssiNG FrAGiLitY iNdiCEs 3

1 Png e pblem agly 5

1.1 Denitions o ragility 51.2 Fragility as a global threat 61.3 Violent confict: cause, symptom or consequence o ragility? 71.4 Why measure ragility? 8

2 Pcng c-cny agly nce 112.1 Background concepts: Recognizing a basic understanding 13

2.2 Systematized concepts: Dening relevant attributes 142.3 Selection and measurement o indicators: Obtaining data 14

2.4 Calculation o index scores: Quantiying the concept 172.5 Presentation o results: Visualizing the numbers 19

3 Cmpang exng c-cny agly nce 23

3.1 Background concepts: What role or producers interests? 233.2 Systematized concepts: What dimensions are included? 25

3.3 Selection and measurement o indicators: Which data sources? 263.4 Calculation o index scores: Do the results dier? 283.5 Presentation o results: How are they visualized? 31

4 selecng an applyng c-cny agly nce 354.1 Using ragility indices: What is possible? 354.2 Selecting ragility indices: What are their relative strengths? 364.3 Five principles or applying ragility indices 37

PArt ii: A CAtALoGuE oF iNdiCEs oN FrAGiLitY 39

te ce e caalge 41Bertelsmann Transormation Index (BTI) State Weakness Index 43Country Indicators or Foreign Policy (CIFP) Fragility Index 47Country Policy and Institutional Assessment (CPIA) /

International Development Association (IDA) Resource Allocation Index (IRAI) 50Failed States Index 53Global Peace Index 57

Contents


6/152

Harvard Kennedy School Index o Arican Governance 60Index o State Weakness in the Developing World 63

Peace and Confict Instability Ledger 66Political Instability Index 70

State Fragility Index 73World Governance Indicators (WGI) Political Stability and Absence o Violence 76

ANNEXEs 79

Annex I: Indicators and data sources used by ragility indices 81Annex II: Aggregation methods used in ragility indices 107

Annex III: List o sources not included in the Users Guide 109Annex IV: A catalogue o ragility and confict qualitative methodologies 113Annex V: Scores o the BTI indicators or identiying state weakness, 2008 121

Annex VI: Technical glossary 125

Enne 129

reeence 135

List oF BoXEs, FiGurEs ANd tABLEs

BxeBox 1: Users o the Country Policy and Institutional Assessment (CPIA) / IDA Resource Allocation Index (IRAI) 8Box 2: OECD 2008 Annual Report on Resource Flows to Fragile and Confict-Aected States 8Box 3: Implications o measurement error: the Peace and Confict Instability Ledger 13

Box 4: Dierent operationalizations o the same concept 14Box 5: Validity and reliability problems in expert surveys 16

Box 6: Tax ratio: a proxy or state ragility? 17Box 7: The pretence o precision: reporting too many digits 18Box 8: Truncated score distributions 19

Box 9: The impression o equidistance in simple result tables 20Box 10: Pitalls o categorization 20Box 11: Mapping ragility: Two visualisations o the Failed States Index 21

Box 12: Comparing scores the case o Bolivia 30

FgeFigure 1: Stages o constructing ragility indices 12Figure 2: CIFP Fragility Index authority, legitimacy and capacity scores or Yemen and Nepal 26

Figure 3: The network o ragility indices and their sources 27

table Table 1: Cross-country ragility indices covered in the Users Guide 2 Table 2: Producers o ragility indices 2

Table 3: Conceptual dimensions covered by ragility indices 25 Table 4: How similar are index results? Bivariate correlations 29 Table 5: 2008 worst country rankings 3

Table 6: Categorization methods employed by ragility indices 32 Table 7: The relative perormance o ragility indices 36


7/152

List of AbbreviAtions*

Bti Bertelsmann Transormation Index

Bti-sW Bertelsmann Transormation Index State Weakness Index

CAst Confict Assessment System Tool

CiFP Country Indicators or Foreign Policy

CiFP-Fi Country Indicators or Foreign Policy - Fragility Index

CPiA Country Policy and Institutional Assessment

CsP Center or Systemic Peace

dAC Development Assistance Committee, OECD

dFid Department or International Development, UK

diE German Development Institute / Deutsches Institut r Entwicklungspolitik

Eiu Economist Intelligence Unit

FAo United Nations Food and Agriculture Organization

Fsi Failed States Index

GPi Global Peace Index

iAG Index o Arican Governance

idA International Development Association

irAi IDA Resource Allocation Index

isW Index o State Weakness in the Developing World

NGo Non-governmental organization

oECd Organisation or Economic Co-operation and Development

PCiL Peace and Confict Instability Ledger

Pii Political Instability Index

PitF Political Instability Task Force

sFi State Fragility Index

uCdP Uppsala Confict Data Program

uN United Nations

uNdP United Nations Development ProgrammeuNEsCo United Nations Educational, Scientic and Cultural Organization

uNhCr Oce o the United Nations High Commissioner or Reugees

uNiFEM United Nations Development Fund or Women

usAid United States Agency or International Development

WGi World Governance Indicators

WGi-Ps World Governance Indicators - Political Stability and Absence o Violence

Who United Nations World Health Organization

l v

*Abbreviations used in the annexes are not listed.


8/152

u gd M F

ACknowLedGments

UNDP and the German Development Institute / Deutsches Institut r Entwicklungspolitik (DIE) acknowledge

with great appreciation the valuable comments received rom the ollowing colleagues and experts making up the

reader group or this publication: Mariano Aguirre (Norwegian Peacebuilding Centre), Louise Anten (The Netherlands

Institute o International Relations Clingendael), Christiane Arndt (Organisation or Economic Co-operation and

Development), Alexander Bellamy (University o Queensland), Felix S. Bethke (University o Duisburg- Essen), Stephen Brown

(University o Ottawa), Diana Chigas (CDA Collaborative Learning Projects), Tobias Debiel (University o Duisburg-Essen),

Michael Frahm (Federal Ministry or Economic Cooperation and Development, Germany), Jrn Grvingholt

(DIE), Pamela Jawad (GTZ), Stephan Massing (Organisation or Economic Co-operationand Development), Celine Moyroud

(UNDP Bureau or Crisis Prevention and Recovery), Eugenia Piza-Lopez (UNDP Bureau or Crisis Prevention and

Recovery), Timothy Sisk (University o Denver), Svein Erik Stave (FAFO), Camilla Sugden (UK Department or International

Development) and Thomas Wollnik (InWEnt). A note o gratitude goes also to Marie Laberge and Ingvild ia (both with

the UNDP Oslo Governance Centre).

We are also grateul or the peer reviews conducted by Hans-Joachim Lauth (University o Wrzburg) and Gerardo Munck

(University o Southern Caliornia).

This publication also beneted rom the language edits done by Alexandra Wilde and Jane Thompson.

Finally, we are indebted to the producers o ragility indices with whom we communicated along the process o developing

this guide to obtain their insights. These include Pauline H. Baker (The Fund or Peace), Martin Brusis (University o Munich),

David Carment (Carleton University), Rachel M. Gisselquist (Harvard University), J. Joseph Hewitt (University o Maryland),

Aart Kraay ( World Bank), Monty G. Marshall (George Mason University), Robert I. Rotberg (Harvard University), YiagadeesenSamy (Carleton University), Camilla Schippa (Institute or Economics and Peace) and Peter Thiery (University o Munich).

Fnng ge wa pve by e


9/152

foreword

There is growing recognition and understanding o the close and maniold linkages between governance and ragility.

At the same time, violent conficts are requently seen as causes, consequences or symptoms o poor, illegitimate and

corrupt governance structures and processes.

Over the past years we have witnessed a marked increase in the attention being paid to situations o ragility their causes,

impact and potential remedies. As a response to this widespread interest amongst development and security actors,

researchers and policy makers, there has also been a sharp increase in the production o various indices which rank

countries according to levels o ragility. The indices refect a broad range o interests, understanding and aspirations

including the larger aid eectiveness agenda.

Despite the prolieration and growing reerence to these indices, no systematic analysis o such indices has been produced

so ar. The Users Guide on Measuring Fragilityattempts to ll this gap by providing a comparative analysis o eleven widely

quoted and used ragility indices. This Guide unpacks the concepts and methods that lie behind the ragility rankings.

This publication is a new addition to a series o users guides published by the UNDP Oslo Governance Centre (OGC) since

2003. As part o the Centres fagship programme on national governance assessment, these guides provide a systematic yet

easy-to-grasp scrutiny o existing indices and indicators through the lens o their potential and current users.

I hope that this Users Guide on Measuring Fragilityserves to provide the reader with guidance on where to nd and how

to use ragility indices, while also stimulating a critical discussion on ragility and governance and how to move orward

towards the development o country-led analyses.

Bjrn Frde, Director

UNDP Oslo Governance CentreDemocratic Governance Group

Bureau or Development Policy

Fwd


10/152

u gd M F

foreword

State ragility has become a buzzword in international development policy. The re-emergence o the state as a central

actor in developing countries has several causes: state ragility is closely linked with security issues at the top o the

oreign policy agendas o donor countries; the current international nancial crisis has made it clear that economic

development and eorts to strengthen markets need eective states; and, last but not least, there is a growing recognition

that accelerating climate change may translate into a prolieration o state ragility in vulnerable developing regions.

While both research and policy are progressing towards a better understanding o ragility, many issues remain unresolved.

One such is the question o how to measure ragility. Valid and reliable indicators are indispensable or improving research

on state ragility, or rethinking political strategies to ameliorate state perormance, and or enhancing the evaluation o

international cooperation with and in ragile states.

Even though scholars have sought to achieve a better understanding o the causes and consequences o state ragility or

some time now, cross-national evidence remains sparse. How ragile would a state have to be in order to prevent successul

democratization? At what level o state ragility is the probability o an outbreak o violent confict signicantly increased?

Through which channels might environmental stress, driven by climate change and the erosion o ecosystems, cause

insecurity and conficts?

Measurement is a necessary prerequisite or the large-scale evaluation and monitoring o interventions related to

ragility. Does state building work? Did (possibly successul) peacebuilding delay or impede the establishment o sel-

supporting state structures? The concepts o results-oriented development policy and o aid eectiveness do not make

any sense without reliable indicators and data.

The areas o research mentioned above are core topics covered by the German Development Institute. Thus, the

institute embarked on this joint project with the UNDP Oslo Governance Centre to study indicators o ragility. The Federal

Ministry or Economic Cooperation and Development (BMZ), whose position on ragility is laid down in its strategy on

Development-oriented transormation in conditions o ragile statehood and poor government perormance, kindly

provided the necessary unds.

The publication at hand is a timely undertaking that will hopeully make political ragility indices more accessible to

development and security experts who are not necessarily experts in statistics. It provides a comprehensive overview o

existing cross-country indices measuring ragility and demonstrates how to use them.

This guide is not a nal but a rst step in understanding and measuring the dynamics o state ragility. While it enables users

to better employ what is already there, the quest or better data in development studies has just begun.

Dirk Messner, Director

German Development Institute / Deutsches Institut r Entwicklungspolitik (DIE)


11/152

This Users Guide on Measuring Fragility presents a comparative analysis o cross-country ragility indices. It assesses their

conceptual premises, methodological approach and possible uses.

The interest in understanding and predicting situations o ragility has grown exponentially amongst research and policy

communities in the last years, in parallel to debates around poor governance perormance, development challenges and

aid eectiveness. As a response to this interest, various ragility indices are periodically published, refecting a diverse range

o interests, purposes and aspirations. Despite the prolieration and ever-increasing use o and reerence to these indices, to

date no systematic, comprehensive study o such indices has been produced.

This Users Guide provides readers with a rigorous, comprehensible and user-riendly examination o country-level

indices measuring acets o ragility. Although there is no common, undisputed denition o ragility, a country could be

said to be ragile when it suers rom a weakness or a ailure in one or several central attributes o the state such as its

eectiveness in providing services to citizens, its authority (including a legitimate monopoly on the use o violence) and

legitimacy. Fragility oten also relates to one or more specic sectors, i.e. security, economic, political or social/cultural,

environmental. The ragility indices in the Guide directly address many o these aspects. It is aimed at empowering the user

with greater knowledge and critical understanding o the subject matter, addressing key questions such as:

What ragility indices are there?

What concepts do they intend to measure?

How well do they measure these concepts?

How should ragility indices be applied?

The intended audience o the Users Guide is current or potential users o ragility indices, especially researchers and

policy-makers working in the area o ragility, governance and confict. Whereas the ormer may nd the guide helpul when

considering ragility indices to inorm their studies, the latter may discover a tool o relevance or cross-national assessments

and impact analysis. In addition, other audiences such as development practitioners or humanitarian NGO workers may

nd some o the debates and ndings rom the Users Guide (e.g. on measurement types and data sources) useul in their

proessional practice.

The Guide includes a selection o 11 ragility and confict indices based on the ollowing criteria: 1

(1) Relevancy:The index has an evident ocus on measuring ragility at the country level.

(2) Quantifcation: The index provides numerical scores on states and is thus potentially suited or cross-country

comparisons.

(3) Accessibility:The index is available ree o charge on the internet in English. 2

(4) Transparency:The index provides inormation about its methodology.

(5) Multi-country coverage:The index provides data or at least 75 countries, or or most countries in a specic region.

(6) Updated inormation:The source is updated periodically, with the latest scores published within the last two years.

introdUCtion:

AboUt tHis GUide

idc:a gd


12/152

u gd M F

This Guide is inormed by a desk review o state-o-the-art research and policy debate and tools on measuring situations o

ragility by quantitative means. In addition, the mapping, selection and analysis o ragility indices were supplemented by

in-person, phone and email interviews with the producers o such indices. 5

The Users Guide is organized in the ollowing manner:

Part Iserves as an introduction to measuring ragility. It is divided into our chapters. The rst chapter covers characterizations

o ragility; the relevance o ragility to, and linkages with, violent confict; and applications o quantitative ragility analyses.

The second chapter explores how to build quantitative, cross-country measures o ragility, uncovering the main eatures,

challenges and pitalls present in each o its ve main stages (i.e. the background concept, the systematized concept, the

selection and measurement o indicators, the calculation o index scores, and the presentation o results). The third chapter

provides a comparative analysis o ragility indices, examining each stage in the building o these indices. Finally, the ourth

chapter gives the reader guidance on how to select and apply ragility indices.

Part IIpresents a catalogue o ragility indices, providing publication details and in-depth inormation on the properties o

each index. The analysis leads to an outline o the indexs strengths and weaknesses as well as its recommended use.

Annex I lists the indicators and data sources used by producers in constructing ragility indices. Annex II gives an overview

o aggregation methods used in ragility indices. Annex III lists quantitative ragility sources not included in the Users Guide

and the main reason or their exclusion. Annex IV provides a catalogue o qualitative assessment tools, which constitute an

alternative inormation source on ragility. Annex V provides the scores o the BTI State Weakness Index, since these scores

are not reported by Bertelsmann. Annex VI is a technical glossary explaining important terms.

table 1: C-cny agly nce cvee n e ue Ge

inex Pce Ang nn

Bertelsmann Transormation Index State Weakness Index Bertelsmann Stitung Bertelsmann Stitung / Center or Applied

Policy Research (Munich University)

Country Indicators or Foreign Polic y Fragility Index Carleton University Norman Paterson School o International

Aairs (Carleton University)

Country Policy and Institutional Assessment (CPIA) /

International Development Association (IDA) Resource

Allocation Index (IRAI)

The World Bank The World Bank

Failed States Index Fund or Peace Fund or Peace3

Global Peace Index Institute or Economics and Peace Economist Intelligence Unit, with guidance

rom an international panel o experts

Harvard Kennedy School Index o Arican Governance

4

Harvard University Kennedy School o Government (HarvardUniversity)

Index o State Weakness in the Developing World Brook ings Institution Brookings Institution / Center or Global

Development

Peace and Confict Instability Ledger University o Maryland Center or International Development

and Confict Management (University

o Maryland)

Political Instability Index The Economist Group Economist Intelligence Unit

State Fragility Index George Mason University Center or Global Policy (George Mason

University)

World Governance Indicators, Political Stability

and Absence o Violence

The World Bank The World Bank Institute

Table 1 below provides an overview o the indices.


13/152

Pa

assessing

Fragility inDiCes


14/152


15/152

.. definitions of frAGiLity

Fragility is a complex and multiaceted concept. There is not as yet an internationally accepted denition o ragility and researcher

practitioners and policy makers alike conceptualize it in dierent ways. There is, however, some consensus within the policy and dono

communities around the OECD denition o ragile states expressed in the Principles or Good International Engagement in Fragil

States and Situations:

States are ragile when state structures lack political will and/or capacity to provide the basic unctions needed or poverty reduction,

development and to saeguard the security and human rights o their populations.6

The Oxord English Dictionarydenes ragile as easily broken or damaged or delicate and vulnerable. Thus, when encountering the

term ragility, the rst question that arises is: ragility o what? In the realm o development policy, two dierent entities are reerred to

as ragile: states and their institutions on the one hand, and societies as a whole on the other.

When ragility reers to the state, ragility is in act a property o the political system. A ragile state is incapable o ullling it

responsibility as a provider o basic services and public goods, which in turn undermines its legitimacy. This has consequences osociety as a whole, threatening livelihoods, increasing economic downturn and other crises which aect human security and the

likelihood o armed confict. In this sense, such phenomena constitute consequences o ragility.

When ragility reers to society as a whole, violent confict and other human-made crises constitute ragility itsel. In this sense, ragilit

is a property o society and thus, being dened much more broadly, includes any kind o political, social or economic instability. Thi

understanding o ragility is termed a ragile social situation.

In this discussion it is crucial to remember that ragility is not tackled in binary terms (all or nothing) but rather as a continuum

that is, a quality that can be present to a greater or lesser degree (i.e. rom high resilience to extreme ailure). In this regard, nationall

led state-building processes o moving towards resilience are the core o the current international agenda, which emphasizes tha

the state-society relations are the centre o gravity o a resilient state7. Furthermore, as we will see, ragility is composed o severa

dimensions, some o which may be more critical than others. In this sense, ragility is not an exclusive property o developing countrie

but can also be ound in many orms and degrees in developed countries. The recognition o this gradation allows or the creation o

indices o ragility, assigning comparable scores to several countries.

The development and research communities have proposed a multitude o denitions o a ragile state that urther blur th

denitional consensus. Moreover, most publications use the term ragile state even when reerring to a broader ragile social situation

Some illustrative examples o denitions o ragility are as ollows:

1. Posing the ProbleM

oF Fragility


16/152

u gd M F

DFIDs working denition o ragile states covers those where the government cannot or will not deliver core unctions to

the majority o its people, including the poor. [] DFID does not limit its denition o ragile states to those aected by

confict. (DFID 2005: 7)

USAID uses the term ragile states to reer generally to a broad range o ailing, ailed, and recovering states. []

the strategy distinguishes between ragile states that are vulnerable rom those that are already in crisis. (USAID 2005:1)

A ragile state [is] unable to meet its populations expectations or manage changes in expectations and capacity through

the political process []. Questions o legitimacy, in embedded or historical orms, will infuence these expectations, while

perormance against expectations and the quality o participation/the political process will also produce (or reduce)

legitimacy. (OECD 2008a: 16)

Fragile states [are] states that are ailing, or at risk o ailing, with respect to authority, comprehensive service entitlements

or legitimacy. (Stewart and Brown 2009:3)

Fragile states lack the unctional authority to provide basic security within their borders, the institutional capacity toprovide basic social needs or their populations, and/or the political legitimacy to eectively represent their citizens at

home and abroad. (Country Indicators or Foreign Policy website, FAQ)

Most o these characterizations implicitly understand ragility as a continuum. Moreover, what these denitions have in

common is that they include one or more central attributes o the state such as:

Eectiveness (how well state unctions are perormed)

Authority(understood as the enorcement o a monopoly on the legitimate use o orce)

Legitimacy(public, non-coercive acceptance o the state)8

Such general attributes are dicult to measure directly. It is thereore necessary to enter into a second level o

measurement, ocusing on indicators o ragility o some or all o these three dimensions. For example, undernourishmento the population or national literacy may provide inormation on the eectiveness o a state, while levels o criminality or

state control over its territory reer to authority. Similarly, the existence or absence o ree, air and regular electoral processes

or revolutions may be indicators o legitimacy.

.. frAGiLity As A GLobAL tHreAt

The term ragile state coexists with conceptually similar notions like weak state, ailing state, ailed state or collapsed state,

all o which may be dened as dierent stages along the ragility spectrum.9 This prolieration o adjectives during the

last decade runs in parallel with renewed and reinorced development and security agendas. Regarding the latter, saving

ailed states10 like Haiti and Somalia in the early 1990s was a rather new issue on the post-Cold War agenda, even though

research had already dealt with implications o weak statehood beore.11 It was not until the terrorist attacks o September

11, 2001, however, that ailed states became a top priority in world politics.12 As or the development agenda, the realization

o the specic challenges arising in ragile states and their impact on human development and poverty eradication eorts

led to context-specic strategies and policies among donors such as the above-mentioned OECD principles or good

international engagement in ragile states and situations. The need or context-tailored development assistance becomes

evident when analysing progress made towards reaching the Millennium Development Goals, with ragile states alling

behind other developing countries.13


17/152

Part 1:a dc

Today, ragile states are seen as the core o many internal and regional development problems as well as security threats

to other states and the stability o the international order.14 Although the understanding o the security threats posed by

ragile states is still highly hypothetical and merits urther investigation, it is oten voiced that ragile states are an ideal

breeding ground or national and international terrorism, organized crime (e.g. human and drugs tracking) and armed

confict. All o these all within the category o asymmetric violent confict that has been termed new wars, 16 related

somehow to state ragility.

.3 vioLent ConfLiCt: CAUse, symPtom or ConseqUenCe of frAGiLity?

Violent confict may be conceptualized as a cause, a symptom or a consequence o ragility, which explains why it is a

dimension o most indices o ragile situations. State ailure may lead to civil unrest, communal violence and armed confict.

When the state does not deliver the basic services it is supposed to, when its authority is limited or arbitrarily exercised, or its

legitimacy systematically questioned, the social contract and public trust weaken to the point where public dissatisaction

easily transorms into violent contestation by sectors o society. In an attempt to regain order, the state oten responds with

violence to the violence caused by its own ailures.

Violent confict and ragility uel each other. State eectiveness, authority and legitimacy are weakened by the highly

damaging eects o violent confict and in extreme situations ragility will maniest itsel in, or contribute to, violent

confict.

Violent confict tends to bring about more violent confict, that is, the likelihood o armed confict is higher when previous

armed conficts have occurred.17 There is little doubt that armed confict has a strong destabilizing eect on states, creating

situations o ragility.

Quantitative ragility measures oten use armed confict databases that have been produced in recent decades to assess

the existence and intensity o interstate and intrastate armed conficts. The denition o armed confict will, o course,determine whether an event is included in the database or not, and thereore the subsequent impact on a given ragility index.

Probably the most used operational denition o an armed confict is the one provided by the Uppsala Confict Data

Program (UCDP):

Armed confict is a contested incompatibility that concerns government and/or territory where the use o armed orce

between two parties, o which at least one is the government o a state, results in at least 25 battle-related deaths in one

calendar year.18

The intensity o an armed confict dened as battle-related deaths will determine categorizations o the confict

extending to situations o high intensity armed confict amounting to war. The threshold to draw a line between low or medium

intensity armed conficts and wars will also depend on the data collector: or the UCDP, or example, at least 25 but less than

1,000 battle-related deaths in a year are considered a minor armed confict, while at least 1,000 battle-related deaths in a

year are necessary to be considered a war.19

The analysed indices o ragility not only consider the intensity o an armed confict but also a range o other security

indicators such as the existence o reugees and internally displaced people, the level o militarization or the illicit trade and

availability o small arms and light weapons. The combination o various security indicators strengthens the robustness o

a ragility index.


18/152

u gd M F

.4. wHy meAsUre frAGiLity?

The increased importance o the ragile states agenda has demanded indices and other tools to help identiy and monitor

situations o ragility and hence make context-specic responses possible. In order to understand the application o a givenindex, however, it is important to make a distinction between intended and real usage; whereas producers may envision a

particular usage or an index, users may utilize an index or a dierent purpose. Whether each o those uses is valid must be

based on the particular circumstances.

Producers o ragility indices have diverse target audiences, ranging rom governments, civil society, multilateral and

bilateral donors, international lending agencies and the private sector, to the academic and research community and the

media (see Box 1).

Similarly, producers tend to present a range o possible uses or ragility indices, mainly revolving around:

Early warning and early action inormation

Evaluation o interventions

Policy guidance

Public awareness Research

Risk analysis

It is crucial to note, however, that a given index may not live up to the producers expectations; any potential application

has particular quality requirements that may not be met by the index. As will become clear in the remainder o the guide, all

indices have to be used with caution. Any application especially those with direct repercussions on people (e.g. resource

allocation) will have to be preceded by a proound analysis o the suitability o a particular index.

Bx 2: oECd 2008 Annal rep n rece Flw Fagle an Cnc-Aece sae

The list o ragile and confict-aected countries used or the OECD 2008 Annual Report on Resource Flows to Fragile and

Confict-Aected States was drawn up using three ragility indices in combination: the Country Policy and Institutional Assessment

(CPIA), the Index o State Weakness in the Developing World and the Country Indicators or Foreign Policy. This marked a change

compared to previous reports, where the list was drawn rom the CPIA only. According to the authors o the report, the use o two

additional indexes that refect the DAC [OECD Development Assistance Committee] denition o ragility and confict (consideration

o both the capacity and legitimacy o the state, and inclusion o the security dimension) aims to make the list more robust and

consistent with the DACs policy ocus. Those two additional indexes add 10 countries to the 38 countries that are identied solely

on the basis o the CPIA. 22

Bx 1: ue e Cny Plcy an innal Aemen (CPiA) / idA rece Allcan inex (irAi)

Despite being produced by the World Bank or corporate purposes, the CPIA is also used externally (or example, by the

European Commission (2008) in the EU Donor Atlas 2008 to benchmark EU aid to situations o ragility20). Some participants at the

meeting Dialogue on the CPIA and Aid Allocation hosted by the Initiative or Policy Dialogue in April 2007 were surprised to learn

that bilateral aid rom Scandinavian countries, the United Kingdom, Canada, and the Special Partnership or Arica all draw on the

CPIA ratings in allocating aid. Certain components o the CPIA eed into the OECD-DAC Aid Eectiveness rating system as well.

Even where not used explicitly, CPIA rankings serve to signal good perormers to other aid agencies. These external uses seem to

ampliy the impact o the CPIA in international development, making careul consideration o the exercise even more important.21


19/152

Part 1:a dc

The indexs objectives infuence its content, and some purposes are unquestionably harder to achieve than others.

For example, an index aimed at predicting destabilization in a way that is meaningul or policy makers requires the

measurement to be suciently sensitive to register small but signicant changes in a timely manner. In the same way,

statistical models intended to provide valuable ragility and confict early warning will be unable to do so unless

they are produced on a regular basis and adjust to an appropriate timeline long-term orecasting models have the

advantage o adapting to the contextual changes that may occur in the course o time, but are o limited use when it comes to

timely warning and the triggering o early action. Similarly, periodical updates are also critical or the purpose o evaluating

interventions. Only repeated measurements allow or the establishment o a baseline and an analysis o trends. Finally,

ragility indices are limited to countries as their xed unit o analysis; they cannot zoom in (i.e. display any changes beneath

the national level) to monitor specic interventions.

Cape smmay

Fragility is a property that may reer to a variety o objects. In development policy and social sciences, ragility usually reers

to states or societies. Although there is no common, undisputed denition o ragility, the main characterizations include oneor several central attributes o the state (i.e. eectiveness, authority, legitimacy).

Situations o ragility pose a threat to local, regional and global stability.

Violent confict may be seen as a cause, a symptom and a consequence o ragility.

Fragility indices are used by donors, development practitioners and government ocials to guide uture action and evaluate

past engagements; by researchers to investigate causes and consequences o state ragility; and by media and the public to

keep track o r isks to human wellbeing. All these proposed usages have to be scrutinized beore implementation.


20/152


21/152

This chapter explains how ragility is represented in numerical values and how to assess the quality o these numbers. Five steps i

the production o an index are considered (see Figure 1):

(1) Articulate the background concept

(2) Systematize the background concept

(3) Select and measure the indicators

(4) Calculate index scores (including aggregation and weighting methods)

(5) Present the results

Theprocess o producing indices is crucial in that knowledge about all these steps is necessary to judge the quality o an index. The

quality o ragility indices, as or any measurement, is described by two criteria: validity and reliability.

Valy reers to the capacity o an index (or indicator) to adequately represent a concept.

relably reers to the capacity o an index (or indicator) to return the same results in repeated measurements.

While suciently high validity and reliability are easy to achieve in everyday physical measurement (e.g. size o a person, weight o

a product), highly abstract concepts like ragility are hard to measure properly. Depending on the intended area o application o ragility index, it is debatable whether it is at all possible to obtain a result o sucient quality. In this sense, creating an index to selec

country cases or urther in-depth study is an easier aim than quantiying ragility to the degrees o precision necessary or quantitativ

research.

The diculty in measuring abstract concepts that cannot be directly observed is maniest in, or example, attempts to achiev

a valid measurement o the states monopoly on the legitimate use o violence. While it is possible to observe certain traits

that constitute the concept such as the geographical reach o police orces or trust o the population in government, they do no

cover the whole concept. This is why most attempts to measure ragility combine several indicators into one index score. Sinc

there is no consensus on which observable traits to combine when measuring the concept, there can be no solution that is universall

acknowledged as correct. Thereore, some ragility indexes are based on a reductionist/minimalist concept while others are mor

comprehensive.

What happens when a measurement is not perectly valid or reliable?24 This insuciency is termed meaemen e

which is the deviation rom the assumed but unobservable true values. The cause o this deviation can be random or systematic

ranm e occur in any measurement, since it is impossible to control or all variables possibly infuencing a measuremen

process. Thus, random error can be interpreted as the inverse concept o reliability. When, or example, in an opinion poll, the wrong

box in the questionnaire is ticked accidentally, the resulting error can be considered random; it is unpredictable and will aec

the results in both directions in the long run.

2. ProDuCing Cross-Country

Fragility inDiCes


22/152

u gd M F

Systematic errors are non-random: Their deviation rom the true values correlates with a actor that can be determined and

which does not level out over time. This means that in the case o systematic error, the measurement does not represent the

concept it is supposed to do, but a dierent one. Thus, systematic error can be interpreted as the inverse concept o validity.

For example, the attempt to measure state capacity to provide welare by the percentage o households with improved water

supply may be systematically biased i there are countries in which other actors had considerable infuence on the expansion o

this service.

I one or both types o error become too large, the quality o an index will not be sucient to justiably derive knowledge

or operational guidelines. The acceptable limit o measurement error is, however, much more easily reached than assumed

even by articles in leading economic and political science journals.25

There are no clear rules on how to assess reliability and validity or ragility indices or social science data in general. Thus, a

user needs to judge the applicability o an index with regard to its intended application. Two dierent but complementary

approaches to assess the quality o an index exist:

(1) Assessing the internal logic o a measurement process (i.e. concept, derived indicators and methods o aggregation)and

(2) Assessing the scores produced by a measurement process with statistical means.

Chapter 3 applies both approaches to existing ragility indices, using the ve-step ramework presented in the remainder

o this chapter.

Fge 1: sage cncng agly nce

(1) Backgn cncep

(the constellation o meanings and

understandings associated with the concept)

(2) syemaze cncep

(the components o the concept)

(3) selecn an meaemen nca

(the primary data)

(4) Calclan nex ce

(the index values)

(5) Peenan e el

(the visualization o the values)

measurementprocess

Basic gure rom Adcock and Collier (2001); modied by the authors.


23/152

Part 1:a dc

.. bACkGroUnd ConCePts: reCoGnizinG A bAsiC UnderstAndinG

The rst step in assessing an index is to identiy the background concept, which in this case is the basic understanding o

ragility. To correctly interpret an index, it is o the utmost importance to know what background concept the producers

are supposing. This may be quite a challenge, since labels and even descriptions o indices do not always state whether the

index reers to ragile state institutions or ragile societies.

While all steps in constructing ragility indices may be a source o measurement error, an insuciently articulated

background concept is the most dicult to correct and oten the most problematic because o the diculty in reaching

shared meanings. For example, two individuals may be quite clear about what they mean by a certain term and assume

that the other has the same understanding, while this is actually not the case. This scenario is more likely when the concept

is new. Divergent assumptions on the background concept between the producer and the user o an index can result in a

systematically biased application.

What are the most common dierences that may be encountered when interpreting background concepts o ragility?

As noted above, ragility reers mostly to the state. Thus, the understanding o the state underlying an index is crucial or

its interpretation. It is generally agreed that the monopoly on the legitimate use o violence is a core unction o the state.

Beyond that, opinions diverge. As a consequence, one may encounter problems with interpreting measurements

because their background concepts are too broad or too narrow or a certain application. Maximalist denitions

Bx 3: implcan meaemen e: e Peace an Cnc inably Lege

Uncertainty is inherent in all measurements. Only when quantied, however, can the measurement error be visualized. The Peace

and Confict Instability Ledger (PCIL), or example, indicates the measurement error o its scores. As the graph shows, lower and upper

uncertainty boundaries stretch quite ar. The scores produced by PCIL are risk ratios, indicating the probability o state ailure

compared to the OECD average. Considering this degree o measurement error one cannot say or sure whether Brazil is less

confict-prone than Somalia, Bangladesh or Central Arican Republic. The large measurement error o the Democratic Peoples

Republic o Korea illustrates the diculty in assessing closed countries; its risk ratio ranges rom a quite stable 2.6 up to a highly

ragile 16.0.

Democratic PeoplesRepublic o Korea

30

20

10

0

Upper uncertainty

Lower uncertainty

Risk ratio

Turkey Brazil Somalia Bangladesh Central AricanRepublic


24/152

4 u gd M F

include ideas o good governance, democratic rule and extensive public service provision. While these ideals are certainly

desirable rom a policy perspective, they complicate considerably the measurement o the phenomenon the more the state

unctions considered, the greater the variables and interdependencies to be controlled. Minimalist measurements, on the

contrary, may easily oversimpliy the phenomenon and end up excluding elements that are crucial or validly representing a

phenomenon.

.. systemAtized ConCePts: defininG reLevAnt AttribUtes

To move rom an abstract background concept towards an operational one requires identiying the concepts core

attributes. These attributes dene the elements that constitute the state. The resulting denition is termed systematized

concept. Most indices in this guide adopt maximalist denitions and include sectors that matter to state ragility: security,

politics, economy, social welare and, in some cases, the environment. This approach is ounded in the assumption o what

services a state should provide or its citizens beyond the maintenance o a monopoly on violence. It is supposed to adhere

to the rules o good governance, stimulate growth, provide public services and sustainably manage natural resources. Such

a systematized concept with a considerable number o sectors and sub-sectors increases the measurement challengesexponentially.

Another obstacle to dening the systematized concept is the specication o particular attributes. I an attribute is dened

as having specic institutional arrangements providing a certain service, it is not valid or countries in which that same

service is provided by other institutional arrangements. A solution to avoid this problem is to put emphasis on the

unction o the object o interest and not on its peculiar orm in a certain setting. However, state unctions are much harder

to measure than institutions since they cannot be directly observed. Many ragility indices try to circumvent this problem by

relying on outcome indicators, which will be explained in the ollowing sub-chapter.

.3. seLeCtion And meAsUrement of indiCAtors: obtAininG dAtA

Ater having selected the theoretical attributes, indicators that represent these attributes are then required. Producers are

aced with the choice to either select existing data and indicators, or to collect new data and transorm it into indicators. In

both cases, one needs to be aware o the properties o these indicators to assess their validity and reliability. The quality o

indicators is undamental to the quality o an index. Biased data sources produce biased indices. Even when data sources are

o high quality, the selection o those indicators that t best is not a trivial task. Box 4 provides an example o how choices

may dier. Four crucial questions have to be considered when selecting existing indicators or producing new ones:

(1) What exactly does the indicator reer to?

(2) How has the indicator been generated?

(3) What countries and years does the indicator cover?

(4) How big is the time lag o the indicator?

Bx 4: deen peanalzan e ame cncep

The choice o indicators or an index may vary greatly even i the indicators measure the same dimension. For example, the Index

o State Weakness and the State Fragility Index operationalize the economic dimension dierently. Whereas the ormer chooses

ve indicators or its economic basket, including gross national income per capita, gross domestic product growth, income in-

equality, infation and regulatory quality (rom the Worldwide Governance Indicators) as economic indicators, the latter opts or

only three indicators including gross domestic product per capita, gross domestic product growth and share o export trade in

manuactured goods that constitute economic eectiveness and economic legitimacy.


25/152

Part 1:a dc

Indicators used in ragility indices may reer to three dierent phases:

(1) Input indicators (also known as structural/rights/commitment/de jure indicators) reer to the existence and quality o

enabling structural conditions. Input indicators ocus primarily on the legal ramework, institutions and procedures in

place in a given country. The questions posed by these indicators commonly require yes or no answers. Indicators o

this include:

Is there a division o powers (executive, legislative, the judiciary) that guarantees the independence o the dierent

branches o the state?

Ratication o Core International Human Rights Conventions

Existence o regulations and public institutions overseeing public expenditure

Country membership o regional and international organizations

(2) Process indicators (also known as responsibility/de acto indicators) measure eorts made to achieve certain outputs

or outcomes. Indicators o this type include:

Health expenditure as a percentage o GDP

Military expenditure as percentage o GDP

International transers o major conventional weapons Pupil-teacher ratio in primary schools

Number o ex-combatants receiving proessional training

(3) Output indicators (also known as outcome/perormance/de acto indicators) measure results o actions. Indicators

o this type include:

Number o confict-related deaths per year

Unemployment

Violent demonstrations and social unrest

Trade balance percentage o GDP

Incidents o victimization that have been reported to the authorities in any given country

Regarding the generation o data, we distinguish our types relevant or measuring ragility: public statistics, expert data,

opinion polls and content analysis.

Public statistics collected by governments, international organizations and non-government organizations. At rst sight,

they may appear to be the most objective type o data generation. They are, however, like any kind o data, aected by

random and systematic error. An example is the tax ratio reported by the International Monetary Fund. In view o the

statistical capacity in many developing countries, it is highly improbable that tax data reported by ragile states satises data

quality requirements.

The generation oexpert data relies on the assumption that people who are actively in certain processes are capable o

giving exact judgments on these processes (see Box 5). A drawback o this kind o data generation is that most experts are

international specialists with similar academic backgrounds and proessional experience. This inclination is likely to bring

about systematic deviations termed expert bias.

In contrast, opinion polls obtain answers rom a representative sample o the population. One such example is the World

Values Surveyused in the Political Instability Index.

A ourth kind o data generation is by automatically analysing text corpora. This technique, called content analysis, has been

introduced into the domain o ragility indices by the Failed States Index. Using Boolean operations, it extracts key phrases

rom tens o thousands o articles available on the internet.26


26/152

u gd M F

All types o data suer rom a common problem o comparability. While sociology has achieved a high degree o

proessionalism in surveying Western industrialized societies, there are severe obstacles to cross-cultural comparisons on

the macro-level. In ragile states, the challenge o identiying and reaching a representative sample o the population adds to

the problem. Collecting reliable primary data is especially demanding in ragile settings, where actors such as widespread

social mistrust, hidden dynamics and agendas, regime secrecy and lack o inrastructure and capacity seriously hamper any

attempt to gather reliable and representative inormation. When these constraints are not suciently addressed, the overall

quality o the source will be put into question, limiting the ability to draw inerences rom the data itsel.

Diculties in data generation aect not only the validity and reliability o indicators, but also their coverage. Any

ragility index will most probably be conronted with missing data in one or more o these indicators. To maintain a

suciently large sample, indices either impute missing data, that is, estimate missing observations with available ones through

statistical models or expert judgments, or they delete missing observations case-wise, i.e. they calculate overall scores even or

countries with one or more missing indicators. The ormer approach is adopted by the Global Peace Index (through the

Economist Intelligence Unit), the latter is the most common procedure adopted by the Index o State Weakness, the State

Fragility Indexand others. I missing data is imputed, the reliability o an index suers, as values or certain countries rely on

guessing. I missing data is deleted case-wise, the validity o an index suers, as certain attributes considered relevant are notincluded in the overall scoring o some countries.

It is not sucient, however, to ask idata is available. It is as crucial to askwhen data is available. The inormation on how

long it takes providers o data to supply indicators is termed time lag. While all indices necessarily draw on data rom the

past, there may be great dierences in terms o how ar back in the past the data was collected. Inant mortality rates, or

example, are collected much less requently than nancial data. This is again mostly due to problems in data generation.

Inant mortality rates are based on household surveys and thus much more resource intensive than collecting data that is

constantly mapped, as is nancial data.

Bx 5: Valy an elably pblem n expe vey

An example o an attempt to directly measure ragility (drawn rom the Bertelsmann Transormation Index) is the ollowing

question to an expert with possible answers:

To what extent does the states monopoly on the use o orce cover the entire territory?

[]

The states monopoly on the use o orce is established nationwide in principle, but it is threatened (or challenged)

by organizations in territorial enclaves (guerrillas, maas, clans).

The states monopoly on the use o orce is established in key parts o the country, but there are organizations (guerrillas,paramilitaries, clans) able to usurp the states monopoly on the use o orce in large areas o territory. (BTI 2008: 16)

Asked to assign a score with the overall score ranging rom one to ten the expert may encounter several obstacles, or

example: How to dene key parts o the country? Do organizations able to usurp the states monopoly on the use o orce need to

possess just the physical means to control the territory, or is a certain degree o legitimacy required (as is usually associated with

that concept)? And again, what are large areas o the territory the insurgents are active in? A orce with little support in society,

controlling ve percent o the country and three medium sized cities could receive any rating between our and seven when

asking ten experts. What i a state is not conronted by serious competitors, but cannot, at the same time, deploy its police orce

to most o the country or inrastructural and nancial reasons? Even an enquiry to hundreds o experts could not exclude the

possibility that the average score would be biased substantially.


27/152

Part 1:a dc

As indices use dierent types o indicators, the time lag inside one index may vary. Dierent practices o indices to mark

time lags aggravate these disturbances: while the 2006 score o the Index o Arican Governance published in 2008 is based

largely on data rom 2006, the Failed States Index 2006 is based on data rom 2005. Implications o time lags dier. Time lags o

socio-economic data do not matter much when they aect phenomena that change slowly, such as lie expectancy, whereas

the measurement o phenomena that may change quickly, like school enrolment, suers more rom time lag.

.4. CALCULAtion of index sCores: qUAntifyinG tHe ConCePt

Ater obtaining data in the orm o separate indicators, producers need to determine the rules or combining this data into a

single index score. For that purpose, indicators need to be brought to a certain range o values (standardization), combined

by mathematical operators (aggregation) and given a particular impact on the nal score (weighting).

Standardization is the rescaling o indicators so that dierences in original scales (like percentages or currencies) do

not have unwanted weighting eects. Scaling indicators means that their values are transormed to a xed range o

numbers, mostly according to the scale o the nal index. This step is decisive or comparability over time. I possible minima

and maxima are determined on the basis o data rom the current year, they may be dierent in the ollowing year.

Accordingly, all values in between these extremes change, and hence may not be compared with values rom a

dierent year. Time invariant standardizations require constant minima and maxima or standardization. These considerations

assume, however, that indicators themselves are comparable over time. I this is not the case, an index constructed to be time

invariant is de acto time variant.

The process oaggregation is dened as the combination o individual indicators through mathematical operations.

Aggregation is necessary in measuring ragility as there is no single indicator yet that could be used to approximate state

ragility. In other words, there is no valid single proxy or state ragility (see Box 6). As a remedy, producers use various

indicators representing attributes o state ragility and combine them into an index, or a latent variable. Two types o

indices exist:(1) Cmpe nce draw on variables which represent dierent attributes (multi-dimensional). Most ragility measures

produce composite indices, such as the Index o State Weakness and the State Fragility Index. They include, among other

variables, the gross domestic product per capita and inant mortality rates.

(2) Aggegae nce draw on variables which represent only one attribute (one-dimensional). The WGI Political Stability

and Absence o Violence measure is an aggregate index. It uses, inter alia, violent social conficts rom the Institutional

Proles Database and the Political Terror Scale. Both indicators reer to the same dimension: security.

Bx 6: tax a: a pxy ae agly?

The most widely acknowledged single proxy or measuring state capacity is the tax ratio.27 Thus, the tax ratio could be

considered an interesting proxy indicator or the state capacity dimension o state ragility. Twelve ragile and confict-aected states

collect less than 15 percent o their GDP in tax with Aghanistan and Zimbabwe collecting less than seven percent

approximately twenty points less than the average or OECD countries (36.2). On the other hand, resource-rich ragile states such

as Iraq, Angola and Equatorial Guinea collect approximately 35 percent.28 When measuring state ragility by the tax ratio, it is

important to consider that there is seldom reliable data on taxation in those states that are most ragile.


28/152

u gd M F

Choices ostandardization can aect how indicators may or may not be aggregated, since dierent levels o scales

allow dierent mathematical operations. Ordinal scales, or example, cannot be used to calculate averages as the distances

between ordinal points are not necessarily equal (which is a prerequisite or calculating arithmetical averages).

Theoretically derived limitations to be considered include the necessity o certain attributes. I one attribute is considered

to be a necessary condition or a state not to be termed ragile, the lack o that attribute should not be compensable by

other attributes.29 For example, a concept based on the assumption that a state is always ragile when security is lacking

denes security as a necessary condition. Selecting as mean o aggregation the addition o security, economy, politics

and social welare would not be valid, since the other dimensions could partly compensate or a lack o security and lit

the country over the threshold o ragility. A more valid method o aggregation would be to multiply the other dimensions

with security. The score will then always be zero when security is zero and thus satisy the conceptual assumption as

a necessary condition.

In the aggregation process, some indicators may have more o an impact on the nal scores than others. The determination

o the relative impact o indicators on the index score is termed weighting. There are two possibilities to determine weights:by theory or by statistical analysis.

Theoretically based weighting derives the importance o indicators rom the underlying concepts o ragility.

Indicators that are deemed more important than others will be assigned greater weights by the producer.

Statistical analysis lets the data determine the weight. Methods like actor analysis and principal components extract

the importance o individual indicators on an unobservable dimension o interest rom a joint dataset. These

methods, however, are also based on assumptions and they are more dicult to control or non-experts.

The aggregation process produces both usable results and waste, including standard errors o statistical approaches,

calibration o expert data and other kinds o aggregate uncertainties that aect the quality o the scores. Producers should

provide these measures o uncertainty or users to judge how reliable the index is. A common deceptive practice is to

use a large number o decimals in reporting results which implies a precision that cannot be achieved by an index (see

Box 7). Indeed, many decimals are only justied i condence intervals that represent the involved amount o uncertainty are

reported. There are several tests that can be used to assess the quality o index scores,30 such as controlling the density o

the resulting score distribution or truncation (see Box 8).

Bx 7: te peence pecn: epng many g

What users may encounter when dealing with ragility indices are scores speciying our or more digits. The problem is that themore digits are specied, the more precision is implied. A score o 2.857, as given to the Central Arican Republic by the Global

Peace Index, implies that one can distinguish the level o peace o another country at 2.850, which is the Democratic Peoples

Republic o North Korea in this case. This is a dierence o about 0.25 percent an indeensible statement regarding the data

quality o indicators used. One solution to this dilemma is to scale values to a precision that may seem less pretentious, as does

the State Fragility Indexby reporting only values between 0 and 24 with no digits attached. Best practice regarding measurement

precision is to report the level o measurement error which qualies the impression o precision. This is done by the WGI Political

Stability and Absence o Violence and the Peace and Confict Instability Ledger.


29/152

Part 1:a dc

Bx 8: tncae ce bn

Sometimes, measurements produce results that place most observations on one side o the scale. This is a sign that the index is

not capable o representing the concept adequately, since the crowded side o the scale cannot distinguish suciently among

cases. In the case o the Peace and Confict Instability Ledger (PCIL), where the requency distribution o scores is truncated at the

lower end (see above, let gure), the skewed distribution is due to the rare occurrence o political instability. As or the remaining

indices, while not all o them reach a near normal distribution like the CIFP Fragility Index (see above, right gure), none yields

severely skewed results.

40,0

36,0

32,0

28,0

24,0

20,0

16,0

12,0

8,0

4,0

0,0

40

30

20

10

0

6,75

6,50

6,25

6,00

5,75

5,50

5,25

5,00

4,75

4,50

4,25

4,00

3,75

3,50

3,25

3,00

2,75

2,50

20

10

0

Peace an Cnc inably Lege CiFP Fagly inex

.5. PresentAtion of resULts: visUALizinG tHe nUmbers

A nal and oten neglected step in producing an index is the presentation o the resulting scores. Ater calculating index

scores, any way o visualizing these numbers can alter the impression on the reader. Means o visualization include tables,

rankings, categorizations, charts and maps. It is easily ignored that all these elements constitute an interpretation o the

scores rather than an objective display o results. Presentation bias does not need to be intentional, however: it can be easily

introduced by accident. Even a simple table can deceive the viewer (see Box 9).

At rst glance, a table gives the impression o equidistance between ranks: a country appearing in the middle o the table

appears to be hal way between the rst and the last country. Even when knowing that dierence in ragility can, i at all,

only be expressed in the dierence o scores, a viewer can hardly escape this subconscious eect. Rankings bolster this

impression, since they explicitly standardize the distance o adjoining countries to one in rank no matter what the real

distance is in score.

Categorizations divide contingent scores into separate sections. For this purpose, thresholds need to be ound that

constitute the boundaries o these sections. This is done mostly by dividing either the range o ranks or the ranks o scores

into equal parts, usually our equal parts (quartiles) or ve equal parts (quintiles). Setting thresholds by rank xes the

number o countries that all into each category: the number o ragile states remains the same over the years, independent

o the development o scores. Conversely, the score values o thresholds move. Using xed ractions with rank thresholds

enables an assessment o relative ragility and whether an index belongs to, or example, the lowest 20 percent (See Box 10).

Statements on absolute trends are not possible, however, with rank thresholds.


30/152

0 u gd M F

Setting thresholds by scores xes their score values. Such a categorization allows or varying numbers o ragile states.

It presupposes, however, that the index is time invariant; otherwise, seemingly xed score thresholds could not be considered

constant. Constant score thresholds allow or detecting absolute changes. They suer, however, rom lacking justication o

why thresholds should be valid just or being equal ractions o a scale, e.g., 2.5, 5.0 and 7.5. Empirically relevant dierences

could in act lie at the values 3.1, 4.5 and 8.0. In general, score thresholds should be theoretically or empirically grounded.

Charts may include colour coding based on categories. Furthermore, one can manipulate the statement o an index greatly

by changing the scaling o the axis or selecting only a certain time span in a chart, or example.

Bx 10: Pall caegzan

The Failed States Index (FSI) is an example o how arbitrary categorization can mislead users. In its presentation in the Foreign

Policy magazine, the FSI categorizes countries into critical, in danger, borderline, stable, and most stable. A table shows the

top sixty countries with the highest risk. The top twenty countries are critical, the ollowing twenty in danger, no matter what

the scores are. This procedure is misleading in at least two ways: rst, the overall risk o the international system appears to be

constant, as there are always twenty critical states listed. Second, a country with a certain score in one year (Yemen, 95.4 in 2008)

may be termed in danger while a country with a lower score in a previous year had been termed critical (Timor-Leste, 94.9 in

2007), even though scores are intended to be time invariant and thus allow comparison over time.

18 95.7 Lebann

18 95.7 Ngea

20 95.6 s Lanka

21 95.4 Yemen

22 94.5 Nge

23 94.2 Nepal

18 95.3 Epa

19 95.2 Bn

20 94.9 tm-Lee

21 93.6 Nepal

22 93.5 uzbekan

23 93.4 sea LeneSource: Foreign Policy (2008: 67) Source: Foreign Policy (2007: 57)

Bx 9: te mpen eqance n mple el able

The Index o Arican Governance (IAG) presents, as most indices do, a list o countries sorted by index score. At rst, this seems

unproblematic. Any ordinary listing, however, gives the impression o equidistance, as depicted in the bar on the let. The bar on

the right depicts how the real values are distributed, showing that Somalia (18.9) is ar worse o by more than 10 points than

the Democratic Republic o Congo (29.8) directly adjacent in the table and that neither Chad (33.9) and Sudan (34.2) nor Angola

(43.3) and the Central Arican Republic (43.6) are nearly as ar apart rom each other 0.3 points each pair as most other countries

are. All these observations could in theory be made by observing the scores given, but in practice, most humans are not able to

grasp all these dierences in a table comprising 48 items at once.

41 Eritrea 46.5

42 Cte dlvoire 45.5

43 Central Arican Republic 43.6

44 Angola 43.3

45 Sudan 34.2

46 Chad 33.9

47 Congo, Democratic Republic 29.8

48 Somalia 18.9


31/152

Part 1:a dc

Bx 11: Mappng agly: tw valan e Fale sae inex

Arica proves to be the continent with most ailed states as measured by the Failed States Index 2008. Depending on the method

o categorization, however, the overall impression may change: in the let map, Egypt is in warning while Kenya is in alert stage;

in the right map, both are in danger.

The map on the website31 The map in the Foreign Policy article32

Cape smmay

There are ve steps in the production o a ragility index: articulation o the background concept, systematization o the

concept, selection and measurement o indicators, calculation o index scores and presentation o the results.

Background concepts underlie each measurement. They need to be clearly articulated to prevent misinterpretation.

Systematized concepts dene the relevant attributes that need to be measured. These attributes must be derived rom the

underlying background concepts and connect these validly with the indicators.

Indicator selection is crucial or both validity and reliability. The quality o indices and indicators is directly aected by the

quality o data they rely on. Social phenomena may be better understood when dierent types o indicators (e.g. input,

process and output indicators) are used. No method o data generation is immune to random or systematic error. Data

gathering in ragile contexts is subject to multiple and severe challenges.

Calculating indices requires the standardization o indicators, choosing a method o aggregation and determining the weights

o indicators. Standardization determines whether an index is time variant or invariant. Both aggregation and weighting

methods need to be ounded in theory. Fragility is a highly abstract concept, prone to error; i inormation on error levels is

missing, it is prudent to assume high error levels.

The presentation o results may lead to misinterpretations. Even simple means o visualization like tables and maps may

distort index results.

The quality o any measurement procedure depends on its validity (i.e. its capacity to adequately represent the concept it

purports to measure) and reliability (i.e. its capacity to return the same results in repeated measurements). Only when all steps

in the production o an index are checked can the quality o an index be estimated.

Geographical maps oten require the categorization o data and thereore suer the same drawbacks (see Box 11).

They bring about additional problems, however, because the geographical size o countries diers signicantly. Thus i

several countries large in area but low in population receive bad scores (red) and several countries small in area but large in

population receive good scores (green), the resulting map provides a negative impression with large red and small green

areas although the large majority o people could actually be living in countries with low ragility.


32/152


33/152

3. CoMParing existingCross-Country

Fragility inDiCes

How does a given ragility index perorm with regard to other ragility indices? To assess ragility indices appropriately, it is

necessary to examine each step in the production o each index. This section ocuses on 11 ragility and confict indices (see Tables 2

& 3) and explores special challenges or measuring ragility. Each assessment o index quality is a relative judgment, however. It canno

provide inormation about the absolute quality o an index. An absolute judgment is not possible since the quality criterion validity

depends on the purpose o application: a measurement that is valid in one context may not be in another. Thus, this chapter provide

only exemplary results o an analysis o core aspects and detailed prescriptions o implementation have to be derived separately in

each case. Users may draw their own conclusions on the quality o an index or a particular application rom inormation provided in

the catalogue o ragility indices (Part II o this guide), while the concluding Chapter 4 will give a rough overview or orientation on the

relative perormance o indices.

3.. bACkGroUnd ConCePts: wHAt roLe for ProdUCers interests?

A rst and undamental obstacle or obtaining a valid measurement o ragility is achieving clarity about the underlying

background concept. As noted above, the abstract nature o the term ragility is already a source o ambiguity, not to mention tha

some sources may measure ragility without calling it such. Consequently, the background concepts o existing ragility indices vary. democracy crucial or long-term stability? Does service delivery belong to the core tasks o the state, and i yes, which sectors are decisive

In ragility indices, there are quite a ew opinions on these and related questions, although a rather broad denition derived rom th

Western welare state prevails (see Part II or quotes rom the indices).

Why are producers interested in measuring ragility and what are the politics o ragility indices?33 No matter what the claimed

purpose is, the practice o measurement will always contain a normative dimension, and this oundation o values oten stems rom th

producers interests. There is a ne line between explicit value-based indices and implicit or covertly biased indices. While it i

legitimate to transparently dene values and assess their occurrence in practice, it is not so when this intent is concealed. Indices

purporting to measure a seemingly universal phenomenon which might in act be a specic expression o social and historica

developments have the potential to mislead their users and in some situations may be interpreted as an attempt to impose th

demand or a specic institutional setup through the backdoor; it denes a country as an underperormer i it does not adhere to th

rules that are promoted as optimal. It is thereore important to know who produces the index and to examine the indexs underlying

assumptions.

Who is responsible or producing indices o ragility? Generally speaking, there are our kinds o actors producing ragility indice

(see Table 1):

(1) Universities

(2) Think tanks

(3) Media corporations

(4) International organizations


34/152

4 u gd M F

Four indices are produced by universities: Carleton University, Harvard University, University o Maryland and George Mason

University.34 Three indices are produced by think tanks: the Fund or Peace, the Institute or Economics and Peace, and the

Brookings Institution. Two indices are produced by media corporations: Bertelsmann and the Economist Group. Two indices

are produced by the World Bank. Governments do conduct ragility or instability assessments, but they naturally rerain rom

publishing lists ranking their ellow states or even themselves. Still, some ragility indices have been directly or indirectly

supported by governments.35 Geographically, all indices are produced by institutions rom OECD countries: most are

US-based; other indices have their roots in Australia, Canada, Germany and the United Kingdom.

In some instances, unding or the indices does not originate exclusively rom the producer. The university-led State

Fragility Index has recently received support rom private oundations. The Index o Arican Governance was originally

sponsored by the Mo Ibrahim Foundation. The Global Peace Index is sponsored by an individual (Australian businessman Steve

Killelea). The Country Indicators or Foreign Policy Project produces its Fragility Index with unds rom various sources,

including the European Commission, Petro Canada and the Canadian Government. In the remaining cases, under and

producer coincide.

inex Pce Fnng ce Ang nn

Bertelsmann Transormation Index

State Weakness Index

Bertelsmann Stitung Bertelsmann Stitung Bertelsmann Stitung / Center or

Applied Policy Research (Ludwig-

Maximilians-Universitt Mnchen)

Country Indicators or Foreign Policy

Fragility Index

Carleton University Canadian Government,

European Commission,

Petro Canada et al.

Norman Paterson School o

International Aairs (Carleton

University)

Country Policy and Institutional

Assessment (CPIA) / International

Development Association (IDA)

Resource Allocation Index (IRAI)

The World Bank The World Bank The World Bank

Failed States Index Fund or Peace Ploughshares / others Fund or Peace and Foreign Policy

(responsible or the article, not the

index)

Global Peace Index Institute or Economics

and Peace

Steve Killelea The Economist Intelligence Unit, with

guidance rom the GPI International

Panel o Experts

Harvard Kennedy School Index o

Arican Governance

Harvard University World Peace Foundation

(ormerly Mo Ibrahim

Foundation)

Kennedy School o Government

(Harvard University)

Index o State Weakness in the

Developing World

Brookings Institution Brook ings Institution Brookings Institution and the Center

or Global Development

Peace and Confict Instability Ledger University o Maryland University o Maryland Center or International Developmentand Confict Management (University

o Maryland)

Political Instability Index The Economist Group Economist Intelligence Unit Economist Intelligence Unit

State Fragility Index George Mason University George Mason University /

oundations

Center or Global Policy

(George Mason University)

Worldwide Governance Indicators:

Political Stability and Absence o

Violence

The World Bank The World Bank The World Bank Institute, World Bank

table 2: Pce agly nce


35/152

Part 1:a dc

The authors designing the indices may also have aliations dierent rom the producers. All university products rely upon

their own sta. Bertelsmann taps this potential by entrusting Munich University with the production o their index,36 while

the Global Peace Index relies upon academics rom various countries as an advisory board. The data used in that index

is calculated and collected by the Economist Intelligence Unit which also produces its own index, the Political Instability

Index. The Failed States Index is produced by Fund or Peaces own sta, as is the Index o State Weakness by authors rom

Brookings Institution and the Center or Global Development. The Worldwide Governance Indicators are authored at the

World Bank Institute while the Country Policy and Institutional Assessment (CPIA) / IDA Resource Allocation Index (IRAI) is

developed by the World Bank personnel rom the countries, regions and headquarters.

The normative orientation o the producer o an index may have an infuence on the construction o the index

(especially on the background concept and how the concept is systematized), and thus aect the countries scores.

The Bertelsmann Transormation Index Status Index, or example, o which the ragility indicators orm a sub-component,

measures constitutional democracy and socially responsible market economy. Since this goal is made explicit, though, one

can make adjustments or possible bias towards certain orms o government and economy.

To proo any suspicion that an index might be promoting a hidden agenda, one needs to careully review the whole

methodology o that index. Sporadic hints do not suce to prove its inapplicability. It is the producers responsibility,

however, to ensure sucient transparency or users to judge whether an index may be deemed impartial or a certain

application.

3.. systemAtized ConCePts: wHAt dimensions Are inCLUded?

Most o the indices in this guide measure ragility along our dimensions that are dierentiated by sectors: security,

political, economic and social dimensions (see Table 3). Only the Country Indicators or Foreign Policy (CIFP) Fragility Index

includes environment as a distinct sector. Other indices include environmental problems only at the level o sub-categories

(Failed States Index, Index o Arican Governance and Country Policy and Institutional Assessment). WGI Political Stability and

Absence o Violence and the Global Peace Index ocus only on the security sector whereas the Political Instability Index

excludes security and ocuses on political, economic and social actors.

secy Plcal Ecnmc scal Envnmenal

CIFP Fragility Index x x x x x

Index o Arican Governance x x x x

Index o State Weakness x x x x

Peace and Confict Instability Ledger x x x x

Failed States Index x x x x

State Fragility Index x x x x

Country Policy and Institutional

Assessment / IRAI

x x x

Political Instability Index x x x

BTI State Weakness Index x x

Global Peace Index x

WGI Political Stability and Absence

o Violence

x

table 3: Cncepal menn cvee by agly nce


36/152

u gd M F

These dimensions reer to the background concept o the Western welare state, which, over the centuries, came to provide

services in all these sectors, with environmental protection being the most recent addition as a response to new societal

demands. This approach could be seen as problematic by some since it does not allow or alternative views on what a state

should provide or (e.g. religious and spiritual needs).

Concepts not only can be disaggregated by service delivery in certain sectors but also by attributes o government.

The CIFP Fragility Index proposes a ramework using state authority, state legitimacy and state capacity as relevant

attributes o a state (see example in Figure 2). Other indices include these unctions as well, but they subsume them under

one o the sectors described above. Unortunately, the measurement o such highly abstract and not directly observable

(latent) concepts as authority, legitimacy and capacity is much more dicult than measuring service provision. This is why

the CIFP Fragility Indexhas to revert to traditionally available indicators like the quality o democracy as measured by the

Polity-Index and these indicators can oten be culturally biased.

3.3. seLeCtion And meAsUrement of indiCAtors: wHiCH dAtA soUrCes?

Which indicators do ragility indices use to quantiy their systematized concepts? Unortunately, the choice o indicators is

determined not only by theoretical considerations but also by limitations o data availability. Gathering cross-national data

that can be condently compared is an enormous task. Most ava

Users' Guide on Measuring Fragility

Documents

Transcript of Users' Guide on Measuring Fragility