Adapting social analytic tools to policy reforms

Anis A. DaniSocial Development DepartmentMay 3, 2004

Structure of presentation

1. Introduction 2. Generic classification of tools3. Application of selected tools in PSIA

1. Stakeholder Analysis2. Qualitative Social Analysis3. Mixed Method Social Impact Analysis4. Social Risk Analysis

4. Challenges

1. Introduction: Where we areThe Social Analysis Sourcebook released in 2002 designed for appraisal of investment projects http://worldbank.org/socialanalysissourcebookSocial analytic tools applied selectively to policy reforms in the past …

… much more frequently since 2002Deliberate decision to delay sourcebook to capture recent PSIA experienceDonor harmonization: sourcebook on macro-social analysis now under preparation in partnership with DFID, with inputs from GTZ and others

Value added of social analysis

Analysis of interests and influence of different stakeholders helps to understand distributional impacts and effects of political economyExplains how social identify and social relations may affect reform outcomes and impactsAnalysis of informal rules and behaviors helps to understand implementation issues and constraintsEmphasis on PSIA process and dialogue helps to identify bottlenecks and preconditions for ownership of reformsHelps to identify socio-political and institutional risks

Analytical focus vs type of data and analysis

Impact of removal of agricultural subsidies on

production

Institutional economics

Access to assets and services differentiated by

gender or ethnicity

Socio-cultural basis of social exclusion

Qualitative analysis Quantitative analysis

Econ

omic

Soci

al

2. Generic classification of Social Analytic Tools

Stakeholder analysisImpact analysis! Qualitative social analysis ! Mixed methods social impact analysis

Institutional analysis Social risk assessment

Stakeholder analysisUniversal relevance! Likely effects on different social groups! Likely impacts of different interest groups! Likely political economy issues relevant to reform

Method:! Identify key stakeholders affected by reform or

likely to impact reform! Analyze characteristics, interests, and influence! Evaluate their pro- or anti-reform stance and

significance! Measures to create ownership or manage risks! Validate results through triangulation or

stakeholder workshopE-applications: Q-Sort and Think Tools but!!!

Example 1: Pakistan Power Sector Reform objectives

Short-termIncrease generation capacityAttract private sector Eliminate fiscal burdenFinancial liabilitiesProtect the poor

Long-termUnbundling and regulationPrivatizationSector sustainability Increased access and quality of service

Pakistan Power StakeholdersMoF/IFIs: fiscal implicationsEnergy utilities: financial liabilitiesProvinces/pol parties: water/revenue sharePrivate sector investors: profits & riskDomestic consumers: access, price, transaction costs, quality of serviceIndustrial consumers: tariff cross-subsidiesCivil society: environmental & social costs of hydropower, consumer protectionGoP: sector restructuring but protection of vote banks and fertilizer industry

Power sector concernsContingent liabilities from power utilities are Rs 30-50 bn (approx 1% of GDP)! Govt. receivables reached Rs. 19.8 Bn in 2002! FATA/FANA receivables Rs. 13.1 Bn! Line losses high: WAPDA 25%, KESC 40-60%;

despite improvements in WAPDA, power theft estimated at Rs. 24.7 Bn per year

! Free electricity to employees adds to lossCost of power ! Generation mix 70:30, now in favor of thermal! IPP capacity contracts at 6.5c/kwh: Rs. 30 Bn

Power sector resultsInitial focus: impact of tariff increase on the poorFiscal and sectoral reform objectives appear to be at odds, with tariff increases leading to perverse impactsIssues of concern, besides tariff reform:! Access to electricity and natural gas inequitable ! Technical losses due to generation mix, imbalance

between production and T&D, line losses! Structural losses due to tariff structure, poorly

targeted subsidies and power theft! Political economy issues, incl gov’t non-payment of

liabilities, and interest group benefits (FATA, staffs)! Institutional issues: regulation; tariff uniformity

across DISCOs; captive power among large consumers; access and barriers to entry; free rider problem; quality of service

Example 2: Zambia Rural ReformThe reform: Transfer land under customary tenure (94 % of land) to state ownership Method: Combined economic and social tools – Rural Household Model, PPA & Ethnographic analysis Approach: Relied on broad stakeholder consultation and capacity building, comparing results with previous benchmarksDissemination: Used available analytical work and in-country capacity. Findings produced in accessible manner for policy makers

Example

Civil Society

The Chiefs

Landless

Herders

FHHs, CHHs

Minority Ethnic Groups

Zambia National Farmers Union (ZNFU)

Local NGOs

International NGOs

Media

Private Sector

Commercial farmers

Small scale farmers

Surveyors

Lawyers

Foreign Investors

Commercial Banks

Donors

USAID

WB/IMF

DFID, GTZ, EU

Government and State agencies

The President

Ministry of Land

Lands Tribunal

The Judiciary

Office of the Vice-President

Ministry of Legal Affair

Ministry of Local Government and Housing

The City, Municipal and District Councils

Ministry of Agriculture and Cooperatives

Ministry of Works and Supply

Ministry of Commerce, Trade and Industry

Ministry of Tourism, Environment and Natural Resources

Ministry of Finance

Parliament

The Police Force/ Ministry of the Interior

Key stakeholders

1

3

6

2

45

7

8

9

16

18

20

17

19

22

23

24

21

25

27

26

28

29

9

11

13

10

1312

14

15

30

32

31

33

Effect of proposed reform

Infl

uen

ce o

ver

deci

sio

ns

Benefit / Support Harm/

Oppose

Hig

h

Lo

w

Neutral

The Land Reform as Suggested by the Government

1

11 62

45

87

1710

9

3

12

13

14

1516

17

18

19

20

21 22

23

24

25

2627

2829

30 3132

33

BACK

Qualitative Social Analysis

The Zambian study used the following qualitative tools: ! Participatory Poverty Research: 10 communities in

the 4 agro-zones (complementing 10 years of ongoing research)

! Ethnographic tool: Repeated study of Kefa village and compared it against 25 year old study that provided a baseline when a chief ran local affairs; " without a chief ==> rise in crime, and inadequate

dispute resolution and resource management

Qualitative social analysisSocial analysis can be undertaken using qualitative tools alone … usually to complement economic analysisTools include PPA, BA, QIM, focus groups, etc.Can yield results at different levels of aggregation – household, sub-groups, community, regionFor research to influence policy, qualitative or participatory research must! be conducted in sites selected through a sampling

protocol to achieve representativity of the units! ensure methodological rigor through use of

standardised tools, field manual, pretesting, etc.

QSA Example 1: Guyana

Study of the bauxite mining sector reform in the town of Linden, GuyanaExpanded from a modeling exercise (T-21 model) of the economic impact of mining downsizing The QSA consisted of key informant interviews, focus groups and a survey administered to focus group respondents

Guyana mining: method & toolsTwo rounds of focus group discussions.! 8 ‘neighborhood focus groups’ to surface key issues, pretest

the interview guide and train field team ! 22 focus group discussions (8-12 persons each) purposively

sampled for geography, occupation/socio-economic status and gender.

35 In-depth Interviews purposively sampled ! First based on geography, which often would proxy for many

socio-economic characteristic (fishing communities, mining communities etc.)

! To capture social dynamics within communities, the sample included men, women and youth as well as different occupations (farmers, retrenched miners, Govt officials).

Questionnaire-based survey of 200 focus group participants ! information on households, income, occupation, education,

access to resources, etc.

One-off severance payment to mine workers did not address other social impactsFirst order effects included! Loss of subsidies for medical care! Loss of credit lines! Insecure land titling of the homestead where miners lived

(previously held by mining company)Significant second round effects on the community:! Loss of subsidies for electricity and water (mining company

subsidies accounted for more than 0.5 % of GDP)! Reduced overall economic activity (leading to out-migration)! Deteriorating community resources previously provided by mining

company e.g. social infrastructure.Weak mining community with high level of inequality; many young people (67 percent under age 30); serious issues of joblessness,substance abuse, hopelessness and idleness; almost 60 percent unemployment; heavily dependence on remittances and still very much defined by bauxite mining.

Guyana: Key Findings

QSA Example 2: Malawi ADMARC reform

In the context of a proposed reform to privatize 204 rural markets run by ADMARC, the PSIA assessed the role and relative importance of ADMARC markets for vulnerable groups in rural areasThe PSIA used quant & qual analyses undertaken separatelythe quantitative work consisted of regression analysis using data from 1998 household survey (HIS-1) and 2002 panel survey (CPS-4)

Malawi ADMARC reform

Methods for the qualitative social analysis:10 markets selected for qualitative analysis; of which 6 marketsproposed for closure, 4 to continue operatingpurposive sampling based on remoteness and livelihood characteristicsAll 10 areas are remote rural areas with poor infrastructure Participatory rural appraisal, focus group discussions and key informants interviews conducted in two villages for each of the 10 markets

Data collection techniques in the 20 villages sampled:! 40 focus group discussions (half with women groups), ! 53 semi-structured interviews, and ! 44 key informant interviews

Malawi: Key findingsQuantitative results:! IHS-1: Distance from ADMARC markets showed positive

impact on household consumption (up to 20% higher)! Analysis of both rounds indicate improvement in household

welfare between 1997 and 2002 related to changes in size of household and cultivated land area

! Proximity to ADMARC has no impact on household incomeKey findings of the qualitative social analysis:! Private traders are progressively replacing ADMARC – but

mainly in areas with good infrastructure – and mainly for purchase of maize not for selling inputs in rural areas

! Beneficial impact of ADMARC markets more important in remote, rural areas that are under-served by private sector

! Role of ADMARC especially important in maize sales during hungry season – food security: ‘social markets’ needed.

Mixed Methods SIA

Combines and integrates qualitative and quantitative tools e.g. purposive or national household surveys with qualitative and participatory research toolsFocus is less on individual tools and more on how to combine and package them into a strong analytical framework that can be adapted to country and reform contextMixed methods SIA are more suitable for sectoral reforms than for economy-wide or macroeconomic reforms

Mixed Method SIA: Example 1Romania Mining Sector

Reform objective to restructure public sector with 278 mines, 173,000 workers and subsidy burden 1% of GDP in 1996Voluntary redundancies reduced workforce by half but severe social crisis led to need for supportive interventionMixed method SIA in 3 major mining regions! 1200 sample household survey! 24 focus groups! 100 household budget analysis! Key informant interviews

Romania Mining FindingsImpact depends on nature of local economy! Mono-industrial vs multi-industrial region! Immigrant workforce vs local labor resulted in

dependence on wage work vs partial agrarian base! Gender of workforce based on nature of mining

Second order effects on local economy! Decline in economic activity! Increased burden on local governments without

additional resources, and increasing mining debts! Fiscal burden not resolved as of 2004

One-off severance based on assumption of labor market absorption was a mistakeNational social protection program not designed to take account of regional and local variations

SIA Example 2: Albania Municipal Water & Wastewater Reform

Selection of study design: Comparison of two reform models! 4 cities with private operator and management

contract (Durres, Fier, Lezha, Saranda)

! 4 control cities with publicly managed utilities (Vlora,Korca, Lushnja, Gjirokaster)

"Controls selected according to similarity in size, socio-economic conditions

Albania Tools and MethodsTools: Stakeholder Analysis, Institutional Analysis, Social Impact AnalysisMixed method: ! Key-Informant Interviews: 109 total (8 Central Gov’t,

101 Local Gov’t)

! Focus Group Discussions: 32 total (4 per city)

! Household Survey: 663 total (80+ per city)

! 8 Community profiles (1 per city)

Research team:! 15 local researchers, multi-disciplinary (urban

governance, political science, water sector specialist)

Albania SamplingPurposive Sampling: Household Survey, Focus groups! Income distribution (4 identified population groups)! Spatial distribution: inside and outside of network (access)• Participants in metering program implemented during Water

Supply Urgent Rehabilitation Project" Focus groups: additional sampling for ethnic, and gender dimensions

Purposive Sampling: Key-Informant Interviews! Organizational stakeholders (Line ministries, municipalities,

utilities)! Expertise in water sector and decentralization process! Public sector, private sector, civil society (incl. interest

groups)

Albania Choice of MethodsHH Survey - data on ! population, education, employment, income, health! perceptions on water supply and sanitation conditions! key problems with service & institutions ! willingness/ ability to pay for improvements

Key-Informant Interview - insights into ! institutional bottlenecks for inadequate water & wastewater

use, ! perceptions on water supply & sanitation conditions, policies

and current roles and functions of respective institutions ! perceptions on organizational changes that will take place

with reform implementation (privatization/decentralization of management & governance of water supply system).

Albania Choice of methods (cont):Focus Group Discussions – qualitative feedback on! nature of water supply and sanitation services, coping

strategies/ distributional impact (tariffs, access, quality)! Willingness/ ability to

" pay for water supply and sanitation improvements, " share responsibilities in rehabilitation, " participate in project implementation, monitoring and evaluation

! perceptions of environmental/ health issues, and meter installation

! social organization of communitiesCommunity Profiles! Water and sanitation conditions, private & public service

provision ! institutional and organizational setup of local sector, and.! socio-economic situation of population, with special focus on

water and wastewater issues, incl. access to services

Albania Findings: lower collection ratio in private utilities

very poor poor

non poor

44.3

24.3

9.8

63.6

27.4

14.70 10 20

30

40

50

60

70

Percentage of Households that are non-regular payers

Privately Managed Cities

Publicly managed Cities

Albania FindingsFormal network access is linked with poverty and migration! Urbanization peripheral settlements of very poor; in cities with

private mgmt utility 72% very-poor HHs have access compared to 100% under public mgmt

Attitudes towards tariffs linked to consumer perception of quality of service! Very-poor HHs = claim inability to afford bill (55% in private mgt

utility cities, and 66% in public mgt utility cities)! Non-poor HHs = Dissatisfaction with service

Willingness to pay water tariffs is low! Many consumers still perceive water as a public good, 2%

of household income, compared to 6% for electricity! Meters currently installed in 22.8% of HHs in private

mgt utility cities, and 26.1% in public utility cities! Over 80% of consumers prefer installation of meters (of

which 98.7% prefer individual meters)

Mixed Methods: Example 3Kyrgyz Electricity Reform PSIA ! combines qualitative and quantitative approaches! uses both macro and micro-level data:

Tools and Methods:! Qualitative data: discussions with all major

stakeholders as well as focus group discussions with the population to collect information and start dialogue

! Analysis of secondary data:" Household survey" Utility billing records" Utility balance sheets

! Household survey merged with utility data and findings from qualitative research

Kyrgyz: Impact on consumers 1

Electricity consumption downExpenditures on electricity upSubstitution up (coal, wood)Quality down (severe rationing/low voltage)Arrears up (sense of entitlement/but who?)Low level of trust/lack of understanding of reform

Kyrgyz: Impact on consumers 2

0

0.5

1

1.5

2

2.5

3

3.5

4

Q1-00 Q2-00 Q3-00 Q4-00 Q1-01 Q2-01 Q3-01 Q4-01

% o

f inc

ome

on e

lect

rici

ty

Bottom quintile

Top quintile

Kyrgyz: Impact on ESI 1

0

5

10

15

20

25

30

35

40

1990 1994 1995 1996 1997 1998 1999 2000 2001 2002

Per

cent

of g

ener

atio

n

residential sales losses

Kyrgyz: Impact on ESI 2 (residential sector)

Year and variable 1999 2000 2001

Consumption (million kWh) 3,802 4,458 3,328

Billed tariffs (million Som) 531.8 762.7 929.5

Collected tariffs (million Som) 345.9 438.4 523.5

Revenue shortfall (million (Som) 185.9 324.3 406.0

Collection rate (%) 65.0 57.5 56.3

Kyrgyz: Impact of residential subsidieson budget

Lifeline @ 150 kwh

Social protection 50%targeting

Social protection 25%targeting

Total cost (US$ million) 28.3 15.6 14.7

Cost effectiveness (US$ per person) 12.7 14.0 26.4

Percentage of GDP 1.86 1.02 0.96

Risk Analysis: What is it used for?Analyze a range of events or trends that could undermine the goals of a given policy reformRisk Management, not risk reduction. Create policy responses that reduce negative potential negative impact this would include:! Avoiding policy choices with unacceptably high costs! Changing policy or designing complementary measures

that respond to different assumptions than those used in the reform as initially proposed

! Developing ex post contingency plans that respond to developments that undermine policy effectiveness or have highly adverse impacts upon stakeholders

! Setting up indicators for monitoring and clear triggers for activating contingency plans

Social Risk Assessment

Assessment of country risk indicators –compiled in SDStats – to achieve objectivityProcess of engaging stakeholders in the identification, assessment and ranking of risks that could undermine the achievement of policy reform goals

Social Risk Assessment: Categories of Risk

Country: endemic risks arising out of socio-cultural, political and institutional context (e.g., ethnic tension, political instability) that are not unique to reformExogenous: risks that originate outside of a country that can affect the achievement of development objectives (e.g., global economic shock, natural disaster)Institutional: Risks of institutional arrangements not dealt with in the reform that can undermine its objectivesPolitical Economy: Risk of elite capture or opposition by key stakeholdersVulnerability: Risks that groups or individuals affected by reform may become more susceptible to exogenous shocks or lose ability to cope with change

Accountability Indicators SAR 2002

405050Nepal

2543505017.365050Pakistan

29.1740.2947.9265.4565.4516.6733.33Average

3916.6733.33Maldives

5037505083.3366.6750Sri Lanka

255866.677510083.3366.67India

2816.6733.33Bhutan

16.6737255061.115050Bangladesh

16.6716.67Afghanistan

Corruption

Press freedom

Index of law and

order

Bureaucratic quality

Democratic accountabi

lity

Political rights index

Civil liberties

index

Standardized scale (1 Low – 100 High)

Cohesion Indicators SAR 2002Standardized scale (1 Low – 100 High)

Nepal

183.3358.51Pakistan

25.2543.7569.40Average

Maldives

5016.6785.59Sri Lanka

2533.3360.42India

Bhutan

2541.6773.09Bangladesh

Afghanistan

EIU political riskEthnic tensionsViolent conflicts

Client consultation to assess risksRecommended method: Stakeholder consultation session on risk assessment! Brainstorm on perceived risks! Share data on external country indicators! Share data from PSIA

Engage stakeholders in assessment and ranking of risks through:! Articulation of assumptions about events or trends

that should (or should not) hold true ! Judging the likelihood that the assumptions will be

valid or risks will occur! Judging the importance of the assumptions and

risks to the reform

Social Risk Assessment Grid

HighSubstantialModerateLow

TTRILow

TTRIModerate

MPMPRRSubstantial

KMPRRHigh

Like

lihoo

d th

at r

isk

may

occ

ur

Importance of risk to policy outcomesSuggested risk management actions:K = Killer assumption. Scratch the design and start over, because risk is unacceptably high.MP = Modify policy. Take action to anticipate likely risk by changing policy design or introducing complementary measures.T = Triggers. Establish measurable indicators that, upon being reached, trigger changes in policy design or measures to address distribution, compensation, adverse impacts, etc.R = Review and reconsiderI = Ignore

4. Challenges

! Improving methodological rigor" Careful selection of study locations and respondents

following sampling protocol or controlled comparisons" Use of standardized tools and field manuals to ensure

consistency and replicability

! Aligning economic and social analytic tools! Influencing policy

" Translating analytical findings into relevant policy recommendations to inform reform design

" Informing policy debate and policy dialogue" Integrating results into downstream M&E

Useful LinksSocial Development Strategy:http://essd.worldbank.org/essdint.nsf/62ByDocName/SocialDevelopmentStrategy

PSIA website:http://www.worldbank.org/psia

PSIA User’s Guide:http://poverty.worldbank.org/files/12685_PSIA_Users_Guide_-_Complete_-

_High_resolution_-_English_-_May_2003.pdf

PSIA Good practice Note:http://opcs/ps-adjustment/gpnonpsia1231.pdf

Participation good practice note:http://opcs/ps-adjustment/GPN%20Notes/GPN%20Participation%203-21.pdf

SDStats – database of social development indicatorshttp://esd.worldbank.org/sdstats/

Social analysis sourcebookhttp://worldbank.org/socialanalysissourcebook