Adapting social analytic tools to policy reforms
Anis A. DaniSocial Development DepartmentMay 3, 2004
Structure of presentation
1. Introduction 2. Generic classification of tools3. Application of selected tools in PSIA
1. Stakeholder Analysis2. Qualitative Social Analysis3. Mixed Method Social Impact Analysis4. Social Risk Analysis
4. Challenges
1. Introduction: Where we areThe Social Analysis Sourcebook released in 2002 designed for appraisal of investment projects http://worldbank.org/socialanalysissourcebookSocial analytic tools applied selectively to policy reforms in the past …
… much more frequently since 2002Deliberate decision to delay sourcebook to capture recent PSIA experienceDonor harmonization: sourcebook on macro-social analysis now under preparation in partnership with DFID, with inputs from GTZ and others
Value added of social analysis
Analysis of interests and influence of different stakeholders helps to understand distributional impacts and effects of political economyExplains how social identify and social relations may affect reform outcomes and impactsAnalysis of informal rules and behaviors helps to understand implementation issues and constraintsEmphasis on PSIA process and dialogue helps to identify bottlenecks and preconditions for ownership of reformsHelps to identify socio-political and institutional risks
Analytical focus vs type of data and analysis
Impact of removal of agricultural subsidies on
production
Institutional economics
Access to assets and services differentiated by
gender or ethnicity
Socio-cultural basis of social exclusion
Qualitative analysis Quantitative analysis
Econ
omic
Soci
al
2. Generic classification of Social Analytic Tools
Stakeholder analysisImpact analysis! Qualitative social analysis ! Mixed methods social impact analysis
Institutional analysis Social risk assessment
Stakeholder analysisUniversal relevance! Likely effects on different social groups! Likely impacts of different interest groups! Likely political economy issues relevant to reform
Method:! Identify key stakeholders affected by reform or
likely to impact reform! Analyze characteristics, interests, and influence! Evaluate their pro- or anti-reform stance and
significance! Measures to create ownership or manage risks! Validate results through triangulation or
stakeholder workshopE-applications: Q-Sort and Think Tools but!!!
Example 1: Pakistan Power Sector Reform objectives
Short-termIncrease generation capacityAttract private sector Eliminate fiscal burdenFinancial liabilitiesProtect the poor
Long-termUnbundling and regulationPrivatizationSector sustainability Increased access and quality of service
Pakistan Power StakeholdersMoF/IFIs: fiscal implicationsEnergy utilities: financial liabilitiesProvinces/pol parties: water/revenue sharePrivate sector investors: profits & riskDomestic consumers: access, price, transaction costs, quality of serviceIndustrial consumers: tariff cross-subsidiesCivil society: environmental & social costs of hydropower, consumer protectionGoP: sector restructuring but protection of vote banks and fertilizer industry
Power sector concernsContingent liabilities from power utilities are Rs 30-50 bn (approx 1% of GDP)! Govt. receivables reached Rs. 19.8 Bn in 2002! FATA/FANA receivables Rs. 13.1 Bn! Line losses high: WAPDA 25%, KESC 40-60%;
despite improvements in WAPDA, power theft estimated at Rs. 24.7 Bn per year
! Free electricity to employees adds to lossCost of power ! Generation mix 70:30, now in favor of thermal! IPP capacity contracts at 6.5c/kwh: Rs. 30 Bn
Power sector resultsInitial focus: impact of tariff increase on the poorFiscal and sectoral reform objectives appear to be at odds, with tariff increases leading to perverse impactsIssues of concern, besides tariff reform:! Access to electricity and natural gas inequitable ! Technical losses due to generation mix, imbalance
between production and T&D, line losses! Structural losses due to tariff structure, poorly
targeted subsidies and power theft! Political economy issues, incl gov’t non-payment of
liabilities, and interest group benefits (FATA, staffs)! Institutional issues: regulation; tariff uniformity
across DISCOs; captive power among large consumers; access and barriers to entry; free rider problem; quality of service
Example 2: Zambia Rural ReformThe reform: Transfer land under customary tenure (94 % of land) to state ownership Method: Combined economic and social tools – Rural Household Model, PPA & Ethnographic analysis Approach: Relied on broad stakeholder consultation and capacity building, comparing results with previous benchmarksDissemination: Used available analytical work and in-country capacity. Findings produced in accessible manner for policy makers
Example
Civil Society
The Chiefs
Landless
Herders
FHHs, CHHs
Minority Ethnic Groups
Zambia National Farmers Union (ZNFU)
Local NGOs
International NGOs
Media
Private Sector
Commercial farmers
Small scale farmers
Surveyors
Lawyers
Foreign Investors
Commercial Banks
Donors
USAID
WB/IMF
DFID, GTZ, EU
Government and State agencies
The President
Ministry of Land
Lands Tribunal
The Judiciary
Office of the Vice-President
Ministry of Legal Affair
Ministry of Local Government and Housing
The City, Municipal and District Councils
Ministry of Agriculture and Cooperatives
Ministry of Works and Supply
Ministry of Commerce, Trade and Industry
Ministry of Tourism, Environment and Natural Resources
Ministry of Finance
Parliament
The Police Force/ Ministry of the Interior
Key stakeholders
1
3
6
2
45
7
8
9
16
18
20
17
19
22
23
24
21
25
27
26
28
29
9
11
13
10
1312
14
15
30
32
31
33
Effect of proposed reform
Infl
uen
ce o
ver
deci
sio
ns
Benefit / Support Harm/
Oppose
Hig
h
Lo
w
Neutral
The Land Reform as Suggested by the Government
1
11 62
45
87
1710
9
3
12
13
14
1516
17
18
19
20
21 22
23
24
25
2627
2829
30 3132
33
BACK
Qualitative Social Analysis
The Zambian study used the following qualitative tools: ! Participatory Poverty Research: 10 communities in
the 4 agro-zones (complementing 10 years of ongoing research)
! Ethnographic tool: Repeated study of Kefa village and compared it against 25 year old study that provided a baseline when a chief ran local affairs; " without a chief ==> rise in crime, and inadequate
dispute resolution and resource management
Qualitative social analysisSocial analysis can be undertaken using qualitative tools alone … usually to complement economic analysisTools include PPA, BA, QIM, focus groups, etc.Can yield results at different levels of aggregation – household, sub-groups, community, regionFor research to influence policy, qualitative or participatory research must! be conducted in sites selected through a sampling
protocol to achieve representativity of the units! ensure methodological rigor through use of
standardised tools, field manual, pretesting, etc.
QSA Example 1: Guyana
Study of the bauxite mining sector reform in the town of Linden, GuyanaExpanded from a modeling exercise (T-21 model) of the economic impact of mining downsizing The QSA consisted of key informant interviews, focus groups and a survey administered to focus group respondents
Guyana mining: method & toolsTwo rounds of focus group discussions.! 8 ‘neighborhood focus groups’ to surface key issues, pretest
the interview guide and train field team ! 22 focus group discussions (8-12 persons each) purposively
sampled for geography, occupation/socio-economic status and gender.
35 In-depth Interviews purposively sampled ! First based on geography, which often would proxy for many
socio-economic characteristic (fishing communities, mining communities etc.)
! To capture social dynamics within communities, the sample included men, women and youth as well as different occupations (farmers, retrenched miners, Govt officials).
Questionnaire-based survey of 200 focus group participants ! information on households, income, occupation, education,
access to resources, etc.
One-off severance payment to mine workers did not address other social impactsFirst order effects included! Loss of subsidies for medical care! Loss of credit lines! Insecure land titling of the homestead where miners lived
(previously held by mining company)Significant second round effects on the community:! Loss of subsidies for electricity and water (mining company
subsidies accounted for more than 0.5 % of GDP)! Reduced overall economic activity (leading to out-migration)! Deteriorating community resources previously provided by mining
company e.g. social infrastructure.Weak mining community with high level of inequality; many young people (67 percent under age 30); serious issues of joblessness,substance abuse, hopelessness and idleness; almost 60 percent unemployment; heavily dependence on remittances and still very much defined by bauxite mining.
Guyana: Key Findings
QSA Example 2: Malawi ADMARC reform
In the context of a proposed reform to privatize 204 rural markets run by ADMARC, the PSIA assessed the role and relative importance of ADMARC markets for vulnerable groups in rural areasThe PSIA used quant & qual analyses undertaken separatelythe quantitative work consisted of regression analysis using data from 1998 household survey (HIS-1) and 2002 panel survey (CPS-4)
Malawi ADMARC reform
Methods for the qualitative social analysis:10 markets selected for qualitative analysis; of which 6 marketsproposed for closure, 4 to continue operatingpurposive sampling based on remoteness and livelihood characteristicsAll 10 areas are remote rural areas with poor infrastructure Participatory rural appraisal, focus group discussions and key informants interviews conducted in two villages for each of the 10 markets
Data collection techniques in the 20 villages sampled:! 40 focus group discussions (half with women groups), ! 53 semi-structured interviews, and ! 44 key informant interviews
Malawi: Key findingsQuantitative results:! IHS-1: Distance from ADMARC markets showed positive
impact on household consumption (up to 20% higher)! Analysis of both rounds indicate improvement in household
welfare between 1997 and 2002 related to changes in size of household and cultivated land area
! Proximity to ADMARC has no impact on household incomeKey findings of the qualitative social analysis:! Private traders are progressively replacing ADMARC – but
mainly in areas with good infrastructure – and mainly for purchase of maize not for selling inputs in rural areas
! Beneficial impact of ADMARC markets more important in remote, rural areas that are under-served by private sector
! Role of ADMARC especially important in maize sales during hungry season – food security: ‘social markets’ needed.
Mixed Methods SIA
Combines and integrates qualitative and quantitative tools e.g. purposive or national household surveys with qualitative and participatory research toolsFocus is less on individual tools and more on how to combine and package them into a strong analytical framework that can be adapted to country and reform contextMixed methods SIA are more suitable for sectoral reforms than for economy-wide or macroeconomic reforms
Mixed Method SIA: Example 1Romania Mining Sector
Reform objective to restructure public sector with 278 mines, 173,000 workers and subsidy burden 1% of GDP in 1996Voluntary redundancies reduced workforce by half but severe social crisis led to need for supportive interventionMixed method SIA in 3 major mining regions! 1200 sample household survey! 24 focus groups! 100 household budget analysis! Key informant interviews
Romania Mining FindingsImpact depends on nature of local economy! Mono-industrial vs multi-industrial region! Immigrant workforce vs local labor resulted in
dependence on wage work vs partial agrarian base! Gender of workforce based on nature of mining
Second order effects on local economy! Decline in economic activity! Increased burden on local governments without
additional resources, and increasing mining debts! Fiscal burden not resolved as of 2004
One-off severance based on assumption of labor market absorption was a mistakeNational social protection program not designed to take account of regional and local variations
SIA Example 2: Albania Municipal Water & Wastewater Reform
Selection of study design: Comparison of two reform models! 4 cities with private operator and management
contract (Durres, Fier, Lezha, Saranda)
! 4 control cities with publicly managed utilities (Vlora,Korca, Lushnja, Gjirokaster)
"Controls selected according to similarity in size, socio-economic conditions
Albania Tools and MethodsTools: Stakeholder Analysis, Institutional Analysis, Social Impact AnalysisMixed method: ! Key-Informant Interviews: 109 total (8 Central Gov’t,
101 Local Gov’t)
! Focus Group Discussions: 32 total (4 per city)
! Household Survey: 663 total (80+ per city)
! 8 Community profiles (1 per city)
Research team:! 15 local researchers, multi-disciplinary (urban
governance, political science, water sector specialist)
Albania SamplingPurposive Sampling: Household Survey, Focus groups! Income distribution (4 identified population groups)! Spatial distribution: inside and outside of network (access)• Participants in metering program implemented during Water
Supply Urgent Rehabilitation Project" Focus groups: additional sampling for ethnic, and gender dimensions
Purposive Sampling: Key-Informant Interviews! Organizational stakeholders (Line ministries, municipalities,
utilities)! Expertise in water sector and decentralization process! Public sector, private sector, civil society (incl. interest
groups)
Albania Choice of MethodsHH Survey - data on ! population, education, employment, income, health! perceptions on water supply and sanitation conditions! key problems with service & institutions ! willingness/ ability to pay for improvements
Key-Informant Interview - insights into ! institutional bottlenecks for inadequate water & wastewater
use, ! perceptions on water supply & sanitation conditions, policies
and current roles and functions of respective institutions ! perceptions on organizational changes that will take place
with reform implementation (privatization/decentralization of management & governance of water supply system).
Albania Choice of methods (cont):Focus Group Discussions – qualitative feedback on! nature of water supply and sanitation services, coping
strategies/ distributional impact (tariffs, access, quality)! Willingness/ ability to
" pay for water supply and sanitation improvements, " share responsibilities in rehabilitation, " participate in project implementation, monitoring and evaluation
! perceptions of environmental/ health issues, and meter installation
! social organization of communitiesCommunity Profiles! Water and sanitation conditions, private & public service
provision ! institutional and organizational setup of local sector, and.! socio-economic situation of population, with special focus on
water and wastewater issues, incl. access to services
Albania Findings: lower collection ratio in private utilities
very poor poor
non poor
44.3
24.3
9.8
63.6
27.4
14.70 10 20
30
40
50
60
70
Percentage of Households that are non-regular payers
Privately Managed Cities
Publicly managed Cities
Albania FindingsFormal network access is linked with poverty and migration! Urbanization peripheral settlements of very poor; in cities with
private mgmt utility 72% very-poor HHs have access compared to 100% under public mgmt
Attitudes towards tariffs linked to consumer perception of quality of service! Very-poor HHs = claim inability to afford bill (55% in private mgt
utility cities, and 66% in public mgt utility cities)! Non-poor HHs = Dissatisfaction with service
Willingness to pay water tariffs is low! Many consumers still perceive water as a public good, 2%
of household income, compared to 6% for electricity! Meters currently installed in 22.8% of HHs in private
mgt utility cities, and 26.1% in public utility cities! Over 80% of consumers prefer installation of meters (of
which 98.7% prefer individual meters)
Mixed Methods: Example 3Kyrgyz Electricity Reform PSIA ! combines qualitative and quantitative approaches! uses both macro and micro-level data:
Tools and Methods:! Qualitative data: discussions with all major
stakeholders as well as focus group discussions with the population to collect information and start dialogue
! Analysis of secondary data:" Household survey" Utility billing records" Utility balance sheets
! Household survey merged with utility data and findings from qualitative research
Kyrgyz: Impact on consumers 1
Electricity consumption downExpenditures on electricity upSubstitution up (coal, wood)Quality down (severe rationing/low voltage)Arrears up (sense of entitlement/but who?)Low level of trust/lack of understanding of reform
Kyrgyz: Impact on consumers 2
0
0.5
1
1.5
2
2.5
3
3.5
4
Q1-00 Q2-00 Q3-00 Q4-00 Q1-01 Q2-01 Q3-01 Q4-01
% o
f inc
ome
on e
lect
rici
ty
Bottom quintile
Top quintile
Kyrgyz: Impact on ESI 1
0
5
10
15
20
25
30
35
40
1990 1994 1995 1996 1997 1998 1999 2000 2001 2002
Per
cent
of g
ener
atio
n
residential sales losses
Kyrgyz: Impact on ESI 2 (residential sector)
Year and variable 1999 2000 2001
Consumption (million kWh) 3,802 4,458 3,328
Billed tariffs (million Som) 531.8 762.7 929.5
Collected tariffs (million Som) 345.9 438.4 523.5
Revenue shortfall (million (Som) 185.9 324.3 406.0
Collection rate (%) 65.0 57.5 56.3
Kyrgyz: Impact of residential subsidieson budget
Lifeline @ 150 kwh
Social protection 50%targeting
Social protection 25%targeting
Total cost (US$ million) 28.3 15.6 14.7
Cost effectiveness (US$ per person) 12.7 14.0 26.4
Percentage of GDP 1.86 1.02 0.96
Risk Analysis: What is it used for?Analyze a range of events or trends that could undermine the goals of a given policy reformRisk Management, not risk reduction. Create policy responses that reduce negative potential negative impact this would include:! Avoiding policy choices with unacceptably high costs! Changing policy or designing complementary measures
that respond to different assumptions than those used in the reform as initially proposed
! Developing ex post contingency plans that respond to developments that undermine policy effectiveness or have highly adverse impacts upon stakeholders
! Setting up indicators for monitoring and clear triggers for activating contingency plans
Social Risk Assessment
Assessment of country risk indicators –compiled in SDStats – to achieve objectivityProcess of engaging stakeholders in the identification, assessment and ranking of risks that could undermine the achievement of policy reform goals
Social Risk Assessment: Categories of Risk
Country: endemic risks arising out of socio-cultural, political and institutional context (e.g., ethnic tension, political instability) that are not unique to reformExogenous: risks that originate outside of a country that can affect the achievement of development objectives (e.g., global economic shock, natural disaster)Institutional: Risks of institutional arrangements not dealt with in the reform that can undermine its objectivesPolitical Economy: Risk of elite capture or opposition by key stakeholdersVulnerability: Risks that groups or individuals affected by reform may become more susceptible to exogenous shocks or lose ability to cope with change
Accountability Indicators SAR 2002
405050Nepal
2543505017.365050Pakistan
29.1740.2947.9265.4565.4516.6733.33Average
3916.6733.33Maldives
5037505083.3366.6750Sri Lanka
255866.677510083.3366.67India
2816.6733.33Bhutan
16.6737255061.115050Bangladesh
16.6716.67Afghanistan
Corruption
Press freedom
Index of law and
order
Bureaucratic quality
Democratic accountabi
lity
Political rights index
Civil liberties
index
Standardized scale (1 Low – 100 High)
Cohesion Indicators SAR 2002Standardized scale (1 Low – 100 High)
Nepal
183.3358.51Pakistan
25.2543.7569.40Average
Maldives
5016.6785.59Sri Lanka
2533.3360.42India
Bhutan
2541.6773.09Bangladesh
Afghanistan
EIU political riskEthnic tensionsViolent conflicts
Client consultation to assess risksRecommended method: Stakeholder consultation session on risk assessment! Brainstorm on perceived risks! Share data on external country indicators! Share data from PSIA
Engage stakeholders in assessment and ranking of risks through:! Articulation of assumptions about events or trends
that should (or should not) hold true ! Judging the likelihood that the assumptions will be
valid or risks will occur! Judging the importance of the assumptions and
risks to the reform
Social Risk Assessment Grid
HighSubstantialModerateLow
TTRILow
TTRIModerate
MPMPRRSubstantial
KMPRRHigh
Like
lihoo
d th
at r
isk
may
occ
ur
Importance of risk to policy outcomesSuggested risk management actions:K = Killer assumption. Scratch the design and start over, because risk is unacceptably high.MP = Modify policy. Take action to anticipate likely risk by changing policy design or introducing complementary measures.T = Triggers. Establish measurable indicators that, upon being reached, trigger changes in policy design or measures to address distribution, compensation, adverse impacts, etc.R = Review and reconsiderI = Ignore
4. Challenges
! Improving methodological rigor" Careful selection of study locations and respondents
following sampling protocol or controlled comparisons" Use of standardized tools and field manuals to ensure
consistency and replicability
! Aligning economic and social analytic tools! Influencing policy
" Translating analytical findings into relevant policy recommendations to inform reform design
" Informing policy debate and policy dialogue" Integrating results into downstream M&E
Useful LinksSocial Development Strategy:http://essd.worldbank.org/essdint.nsf/62ByDocName/SocialDevelopmentStrategy
PSIA website:http://www.worldbank.org/psia
PSIA User’s Guide:http://poverty.worldbank.org/files/12685_PSIA_Users_Guide_-_Complete_-
_High_resolution_-_English_-_May_2003.pdf
PSIA Good practice Note:http://opcs/ps-adjustment/gpnonpsia1231.pdf
Participation good practice note:http://opcs/ps-adjustment/GPN%20Notes/GPN%20Participation%203-21.pdf
SDStats – database of social development indicatorshttp://esd.worldbank.org/sdstats/
Social analysis sourcebookhttp://worldbank.org/socialanalysissourcebook
Top Related