Design for Evaluation: Gordon & Betty Moore Foundation’s ... · Design for Evaluation: Gordon &...

1

2009 Environmental Evaluators Networking Forum

“Integrating Evaluation into a Program’s Design”

The George Washington University Trachtenberg School of Public Policy and Public Administration

Design for Evaluation:

Gordon & Betty Moore Foundation’s Andes Amazon Initiative

Luis A. Solórzano & Jared Hardner

Washington DC, June 8, 2009

2

Overview of Andes Amazon Initiative

Gordon & Betty Moore Foundation is a science-based, results-driven philanthropic organization. The Foundation operates proactively in three specific areas of focus: environmental conservation, science, and the San Francisco Bay Area. Distinct Initiatives have been created within these three Program areas. An Initiative employs a portfolio of grants that are expected to help achieve targeted, large-scale outcomes in a specific time frame.

Andes Amazon Initiative (AAI) is the largest private donor for conservation in the Andes and Amazon Basin. Its goal is to maintain sufficient natural habitat to preserve climate function and biodiversity in the region. AAI is currently engaged in a refresh of its strategic plan for the coming five years, with an expected budget of approximately $176-180 million.

3

Outcome Statement:Unconstrained

Net loss of forest cover in the Basin is reduced to zero or evenreversed.

Climatic function of the biome is secured by ensuring that 60-80% of forest cover in each ecoregion is maintained in appropriate land uses.

Representative biodiversity at the ecoregional level is maintained by ensuring that at least 10% of each ecoregion is effectively managed as IUCN Category I-III protected areas.

4

Outcome Statements:Constrained 2010-2014

Achieve the unconstrained outcomes for priority landscapes (a subset of ecoregions) within the Basin, while at the same time addressing issues that will benefit conservation across the basin over the longer term.

5

Designing for Evaluation

Through design, we attempt to address the most common problems we encounter in conservation programs. These are problems with design, and subsequently create problems for evaluation of performance:

– Prioritization of resource allocation

– Necessary and sufficient interventions

– Meaningful measures of performanceConcrete outcomesInterim measures of progressProgress against baseline & counterfactualAttribution

6

Plan Structure

Understanding context

7

Plan Structure

Detailed assessment of priority ecoregions and development of strategies

8

Plan Structure

• Final selection of priority landscapes

• Theory of change

• Action plan

• Implementation

9

Plan Structure

• Ongoing revision of targets

• Monitoring and evaluation

• Revise strategies as needed

• Select new geographies based on progress

10

Target Setting

Maintain climatic function– 60-80% forest cover target (each ecoregion)

Maintain representative biodiversity– Default target is 10% of each ecoregion in IUCN Category I-

III Reserves (Convention on Biological Diversity Commitment)

11

Progress towardsTargets

12 Climate Target

Options for achieving targets differ drastically among ecoregions

0

1

2

3

4

5

6

Many Options

Progress Towards Climate Targets

Few Options

No Options (Without Restoration)

13

Deforestation risk varies across basin

0

1

2

3

4

5

6

Vulnerabilityto Deforestation

14

Prioritization

15

Top Ten Ecoregions

16

Limiting Factors

Measures presence/absence of factors that threaten conservation, but vary according to geography– Examples

Economic: Agro-industry expansion, loggingPublic Policy: IIRSA (roads, dams)

Identifies and measures criteria necessary and sufficient to achieve conservation– Example

Strategy to address logging exclusively via Forestry Certification may miss essential complementary issues such as enforcement of environmental laws for those not persuaded by market incentives

Provides common basis to measure progress towards longer term outcomes, such as reduced deforestation

17

Limiting Factors

Example of limiting factor scoring…

pDetection x pProsecution x Penalty > Benefits of illegal activities -- sufficient as deterrent and violations do not occur, enabling conservation

pDetection x pProsecution x Penalty > Benefits of illegal activities -- sufficient as deterrent and violations occur with low frequency, not limiting conservation

pDetection x pProsecution x Penalty < Benefits of illegal activities -- insufficient as deterrent and violations occur with moderate frequency, limiting conservation

pDetection x pProsecution x Penalty = 0 -- no deterrence on illegal behaviorviolations occur with high frequency

Enforcement

-Penalty

1=Enabling0=Not Limiting-1=Limiting-2=PreventsLimiting Factor

18

Limiting Factors Scores

-2

-1

0

1

Pu

blic

Po

licy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g

Public Policy, Legislation,and Regulatory Framework Enforcement Institutional Capacity

EconomicBalance

Stakeholder Engagement& Support Conservation Funding

IUCN CATEGORIES I-III LIMITING FACTORS

19

Which Strategies are Needed?

-2

-1

0

1

Pu

blic

Po

licy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g


EconomicBalance



SE Amazon

Local Strategies:

Planning

Implementation

Stakeholder Engagement

Stakeholder Support

Cost Analysis

Set up Costs

National Strategies:

Prosecution

Penalty

Global Strategies:

Economic incentives

Long-term funding

20

Which Strategies are Needed?

-2

-1

0

1

Pu

bli

c P

oli

cy

Leg

isla

tion

&R

eg

ula

tion

s

Dete

ctio

n

Pro

secu

tion

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

nom

icact

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Cost

An

aly

sis

Don

or

Coord

inati

on

Set-

up

Cost

Fu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lon

gte

rm f

inan

cin

g


EconomicBalance



-2

-1

0

1

Pu

blic

Po

licy

Leg

isla

tio

n &

Reg

ula

tion

s

Dete

ctio

n

Pro

secu

tion

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Cost

An

aly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Cost

Fu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g

Public Policy, Legislation,and Regulatory

Framework Enforcement Institutional CapacityEconomicBalance



SE Amazon

Brazilian Amazon

21

What Strategies are Needed?

-2

-1

0

1

Pu

bli

c P

oli

cy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g


EconomicBalance



-2

-1

0

1

Pu

blic

Po

licy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g


EconomicBalance



SE Amazon

Amazon Basin

22

Who FundsStrategies?

Donor mapping– Donor database(s)

Donor collaboration– Formal vs Informal

Current Grants/Loans Active in Basin at Present (Order of Magnitu

$0.00

$500.00

$1,000.00

$1,500.00

$2,000.00

$2,500.00

$3,000.00

$3,500.00

$4,000.00

$4,500.00

Brazil Colombia Ecuador Peru Bolivia Guyana Suriname Venezuela

Mil

lio

ns

of

US

$

Current Grants/Loans Active in Basin - no Brazil (Order of Magn

$0.00

$50.00

$100.00

$150.00

$200.00

$250.00

$300.00

$350.00

Colombia Ecuador Peru Bolivia Guyana Suriname Venezuela

Mil

lio

ns

of

US

$

23

Who Funds Strategies?

Donor mapping & collaboration– Addressed by others (World Bank & Gov’t of Norway)– Addressed by Moore & others– Unaddressed

-2

-1

0

1

Pu

blic

Po

licy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g


EconomicBalance



24

Who Implements Strategies?

Donor mapping & collaboration– State Government– Federal Government– NGO 1… n

-2

-1

0

1

Pu

blic

Po

licy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g


EconomicBalance



25

Theory of Change

For each priority geography, an explanation of:

– How selected strategies address all relevant limiting factors

– How strategies are funded

– Who implements strategies

26

Monitoring &Evaluation

Outcome Metrics– Net deforestation (target = 0%)– % forest cover (target = 60-80%)– % natural habitat in IUCN Cat I-III (target = 10%)

Interim Metrics– Change in limiting factors (target = TBD by region, nation, basin)

Metrics include 4 elements– Baseline (via land use analysis, LF analysis)– Counterfactual (via land use change models, LF analysis)– Current status (via land use analysis, LF analysis)– Attribution (via donor mapping and systematic collaboration)

27


Interim Metrics: Change in limiting factors

-2

-1

0

1

Pu

bli

c P

oli

cy

Leg

isla

tion

&R

eg

ula

tion

s

Dete

ctio

n

Pro

secu

tion

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

nom

icact

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Cost

An

aly

sis

Don

or

Coord

inati

on

Set-

up

Cost

Fu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lon

gte

rm f

inan

cin

g

Public Policy,Legislation, and

Regulatory Framework Enforcement Institutional CapacityEconomicBalance

StakeholderEngagement & Support Conservation Funding


-2

-1

0

1

Pu

bli

c P

oli

cy

Leg

isla

tion

&R

eg

ula

tion

s

Dete

ctio

n

Pro

secu

tion

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

nom

icact

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Cost

An

aly

sis

Don

or

Coord

inati

on

Set-

up

Cost

Fu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lon

gte

rm f

inan

cin

g





Baseline

(2009)-2

-1

0

1

Pu

bli

c P

oli

cy

Leg

isla

tio

n &

Reg

ula

tio

ns

Dete

ctio

n

Pro

secu

tio

n

Pen

alt

y

Pla

nn

ing

Imp

lem

en

tati

on

Eco

no

mic

act

ivit

ies

com

pati

ble

wit

hco

nse

rvati

on

En

gag

em

en

t

Su

pp

ort

Co

st A

naly

sis

Do

no

rC

oo

rdin

ati

on

Set-

up

Co

stFu

nd

ing

Recu

rren

t C

ost

Fu

nd

ing

/Lo

ng

term

fin

an

cin

g





Counterfactual

(2014)

Actual

(2014)

Attribution:

Major Role

Partial Role

No Role

28


Outcome Metrics:

Forest cover maintained (all land categories)

Habitat conserved (IUCN Cat I-III)

Baseline

(2009)

Counterfactual

(2014)Land use change

modeling

Actual

(2014)

Attribution: Major RolePartial RoleNo Role

29

Concluding Remarks, Questions, Suggestions…

Design for Evaluation: Gordon & Betty Moore Foundation’s ... · Design for Evaluation: Gordon &...

Documents

Transcript of Design for Evaluation: Gordon & Betty Moore Foundation’s ... · Design for Evaluation: Gordon &...