1

IPDET Lunch Presentation Series

Equity-focused evaluation: Opportunities and

challenges

Michael Bamberger June 27, 2013

2

Why equity matters

• Many programs do not reach/benefit the poorest and most vulnerable sectors

• Widely-used indicators [MDGs, HDIs, economic growth] are based on averages and conceal gap between the poorest [bottom 20%] and the rest of the population is not being reduced

• The poorest groups are invisible

• Poor and vulnerable groups have special needs that are not addressed

• Major gaps continue to exist between men and women• No society has achieved full gender equity

3

Defining equity

• Equity refers to fairness

• It examines unequal outcomes (e.g. income, health, access to electricity) to determine if differences are due to: • Unavoidable circumstances (e.g. floods, war, international

economy, limited public resources)

• Free choice (some people may prefer less working hours and more leisure)

• Avoidable circumstances = inequity:

• Unequal opportunities (family background, education, poverty)

• Contextual factors (economic, political, social, environmental etc)

4

Opportunities

• Education• Health• Security• Housing

OutcomesContextual factors

• Economic• Labor markets• Transportation• Political• Socio-cultural• Environmental• Psychological

• Education• Employment• Income• Health and life

expectancy• Political

participation and voice

Unavoidable• Aging• Natural

disasters?• War?• Economic cycles?• Geographic

location?

EQUAL

UNEQUALand

inequitable

UNEQUALBut not

inequitable

Avoidable

Free choice?

UNEQUAL

EQUAL

Linking opportunities and outcomes in equity analysis

+/- effects

on equity

5

Determinants of inequity

• Income

• Location

• Ethnicity, caste, religion

• Gender

• Disability

• Legal systems

6

Why so many evaluations do not address equity

• Political reasons• Equity analysis challenges indicators of progress

(MDGs, HDI etc)

• Governments do not wish to give equal benefits

• Roma, refugees, sexual orientation, ethnic minorities

• Socio-cultural• Society may support marginalizing “undesirable”

groups

• Methodological

7

Methodological challenges for evaluating equity

• Lack of disaggregated data

• Many outcomes are sensitive or difficult to identify and measure

• Unanticipated outcomes

• Complex processes of behavioral change

• Equity evaluation requires new methodologies

• Findings can be threatening

8

Equity focused evaluation

• Assessing the relevance, effectiveness, efficiency, impact and sustainability of interventions on equitable development

• Identifying which groups benefit most and least from programs and policies

• Assessing effectiveness of policy frameworks and identifying policy gaps

9

Some equity-focused evaluation frameworks

• Theory of Change

• Contribution analysis

• Realist evaluation

• Social exclusion analysis

• Gender analysis

• Systems analysis

• Bottleneck analysis

• Policy gap analysis

10

Benefits of systems analysis for equity evaluation

• Programs are embedded in existing social and political systems with historical traditions and linkages among stakeholders

• Different actors have different perspectives on the program

• Boundaries – open or closed

• New interventions cause contradictions and conflict

• Outcomes determined by how conflicts resolved

Bottleneck supply and demand framework: factors affecting use of services by vulnerable

populations

11

Use of services by vulnerable population

Supply side factors

• Coverage •Budgets and available resources

Demand side factors•Knowledge, Attitudes and Practice of vulnerable groups• Community ownership•Culturally acceptable services• Culturally sensitive staff• distance• cost of travel and fees• available transport•time •poverty

Contextual factors

• Economic• Political• Institutional• Legal and administrative• Environmental• Socio-cultural

12

Country and project level tools for equity analysis

• Multi-dimensional poverty index (UNDP)

• Quintile analysis

• Public expenditure incidence analysis

• Gender inequity analysis

• Access to public services [e.g. Citizen Report Cards]

• Participant observation

• Participatory group techniques [e.g. Wealth Ranking]

13

Evaluators’ E quity Checklist

• Who are the poorest and most vulnerable groups in your program? • What constraints affect their participation?

• Does your evaluation address these constraints?

• Does your sampling frame cover all of the target population? Are you sure about this?

• Who does and does not use project services? Why?

• Are there any constraints on women’s participation?

• Will your evaluation design detect unanticipated and negative outcomes? Are you sure about this?

14

A final reality check

• Do you really understand who are the vulnerable and marginal groups?

• Are you really sure their situation will be covered by the evaluation?

• Will your data collection instruments really capture the situation of invisible groups who do not want to be interviewed, who are socially marginalized or in fear of persecution?

15