ADB TA 4581 —Developing a National Result-Based

M &E System for Key Projects

09-2006

Prof. Xue Yan, in behalf of TA 4581

TA 4581 Project Objective

Strengthen capacity of Key Project Inspectorate Office which is under the Ministry of National Development and Reform Committee (KPIO) to develop result-based performance indicators of key national projects.

Background

• KPIO created 1998:– major responsibility to monitor and inspect key

national projects;– first inspections January 1999.

• The magnitude of KPIO’s – >10,000 “national key projects”;– ~1,600 projects inspected up to 2005;– 85 staff (Beijing), ~500 (provincial & municipal

KPIOs).

Background KPIO’ s emphasis to date is on project implementation:

financial control schedule achievement quality of outputs corruption detection

This TA will assist development of results-based indicators to supplement the current system, and raise knowledge and awareness of Results-based monitoring & Evaluation methods. A a more complete picture will be gained, to tell: if the original purpose has been achieved; how well the project was implemented; how to improve operation of the finished project; lessons for future projects.

Project Approach

Test results-based indicators through selected project case studies.

Train KPIO staff and government officials, using case studies.

Integrate selected indicators into KPIO system.

Results-based Monitoring and Evaluation Systems

Results Results The outputs, outcomes and impacts of a project.

Monitoring Monitoring – Monitoring aims to ensure the project is on the right track and

necessary adjustments are made to ensure successful outputs and outcomes. Monitoring also provides the basis for Evaluation, as it demonstrates project progress and points to possible adjustments to project plans and targets.

– Monitoring is continuous from the beginning of a project.

EvaluationEvaluation The systematic and objective assessment of an in-progress or

completed project. Includes assessment of its design, implementation and results.

Results-based Monitoring Model

Strategic goal

Inputs Activities Outputs

Impact

OutcomeNeed

The Model with the Project Cycle

Project Project Implementation Implementation

PhasePhase

Project Project Implementation Implementation

PhasePhase

Strategic goal战略目标

Inputs输入

Activities活动

Outputs提交物

Impact长期效应

Outcome直接结果

Need需求

Strategic goal战略目

Inputs输入

Activities活动

Outputs提交物

Impact长期效应

Outcome直接结果

Need需求

ConceptConcept

FeasibilityFeasibility

DesignDesign

OperationOperation

Bidding & Bidding & ContractingContracting

In the past, project monitoring focused on implementation phase: activities and outputs.

Now governments increasingly focus on outcomes and impacts, to demonstrate value of projects in an accountable & transparent system.

Systems vary from country to country but all developed countries and many developing countries are working in the direction of Result-Based performance monitoring and evaluation.

Inputs

Activities

Outputs

Impact

Outcomes

Project budget.

Major activities carried out to create the project deliverables.

Physical deliverables of the project.

Statement of the long-term or strategic objectives.

Information on the expected direct results of the project being completed and put into operation.

The Framework

The framework summarizes how the project will achieve results by converting inputs to outputs, and how outputs are expected to give the desired results.

R-BR-B(addition)(addition)

NakprosesNakproses(current)(current)

Results-Based Development

Strategic goal

Inputs Activities Outputs

Impact

OutcomeNeed

The interesting parties

Ministry Department KPIOProject sponsor

Optimize performance Minimize waste Public accountability Good governancePerformance culture Benchmarking

State Council FinanceNDRCMinistry Critically evaluate benefits

Prioritize future projectsGood governance

Outputs

Impact

Outcome

Ministry Department KPIO Project sponsor Project manager Contractor

Costs Efficiency Quality Timeliness Funding Corruption Public accountability

Benefits of Results-Based Monitoring

Optimize project outcome performance Avoid mistakes from the past Priorities future project proposals Greater public accountability Can demonstrate good governance Have a performance culture Minimized project waste Set higher standards using Benchmarks

Results Monitoring Model

Develop industry

MaterialsEquipmentManpower

Construct roads New road

Industry grows

Road usage, Speedy, Smooth

Good transport access

Results Monitoring Indicators

New road

Industry grows

Road is used, Fast, Smooth

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

2000 2001 2002 2003

Vehicle users汽车使用者

Results Monitoring Indicators

New road

Industry grows

Road is used, Fast, Smooth

0

10

20

30

40

50

60

2000 2001 2002 2003

Averagetraveltime 平均旅行时间

What we want to know after construction ends and asset is being used

is the finished project being used as intended? is the operating performance as planned or do

we need to make some modifications to make it work better?

is the project achieving the intended results?

What we want to know later (>3 years after construction):

did the project achieve the intended operating results?

did the project achieve the intended impact? should we do similar projects in future? what lessons have we learned from this project

that we can apply to future projects?

Outcomes• Usage rate

– Average number of vehicles per day travelling on road.

– Output of productive unit:• e.g. MWh provided to industry

• Quality– Average travel time between A and B by road– quality of electricity output, e.g. no load shedding, xx

MW peak load maintained

• Financial– revenue from the asset usage (calculated as pay-

back period or rate of return FIRR)

Outcome indicator: Usage

Nanpu BridgeAverage number of vehicles per dayYear Forecast Actual

1992 19424 166121993 33026 199101994 34986 303001995 36848 455761996 35195 564561997 37272 627001998 39471 589001999 41799 600002000 44274 63500

Outcome indicator: UsageAverage number of vehicles per day

Nanpu Bridge

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

1992 1993 1994 1995 1996 1997 1998 1999 2000Year

Ve

hic

les

pe

r d

ay

Forecast

Actual

What does this tell us about the project?

Outcome indicator - Usage

What does this tell us about the project?

Indicator 1

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12

Year after completion

Us

ag

e

Forecast

Actual

Outcome indicator: Operating Revenue

Financial PerformanceYear Forecast Actual

Toll revenues1992 - 16.3 1993 80.8 37.5 1994 84.7 77.3 1995 88.1 138.1 1996 82.9 170.6 1997 87.2 188.3 1998 91.0 175.1

Outcome indicator: Operating Revenue

Toll Revenues - Nanpu Bridge

-20.040.060.080.0

100.0120.0140.0160.0180.0200.0

1992 1993 1994 1995 1996 1997 1998

Year

Y m

illi

on

Forecast

Actual

What does this tell us about the project?

Outcome indicator: Operating profit

Financial PerformanceYear Forecast Actual Forecast Actual

Toll revenues Gross Operating Profit1992 - 16.3 - 9.9 1993 80.8 37.5 72.8 28.2 1994 84.7 77.3 75.7 62.0 1995 88.1 138.1 78.4 123.0 1996 82.9 170.6 72.4 152.0 1997 87.2 188.3 75.9 175.8 1998 91.0 175.1 78.8 163.8

Outcome indicator: Operating profit

Gross operating profit - Nanpu Bridge

-

20.0

40.0

60.0

80.0

100.0

120.0

140.0

160.0

180.0

200.0

1992 1993 1994 1995 1996 1997 1998

Year

Y m

illi

on

Forecast

Actual

What does this tell us about the project?

Case Studies – Objectives• Objectives:

1. Pilot Result-Based performance indicators, to bring international best practices into project monitoring and evaluation

2. Find practical ways of implementing them into the current computer system

3. Develop cases and materials for promotion of and training in Result-Based Monitoring & Evaluation

4. Provide learning experience with Result-Based performance indicators for KPIO, KPIO provincial level and executing agencies

Considerations for case study projects Sectors

Power Transport Hospital Others

Status Commenced project implementation Recently completed Completed 3 – 5 years

Qualities Good or OK Problematic

Desirable variety of case study projects

• Contract types– turn-key – Build-Own-Transfer– multiple contractors/consortium– etc…

• investment sources– Government– International loans– Private or foreign investment

• other– large/small– profit/non-profit

Steps

1. Choose case study projects.2. Gather documentation on project justification 3. Develop monitoring framework for each

project.4. Define results-based indicators for each

project.5. Collect data for result-based indicators for

each project.6. Assess indicators, change if necessary.7. Use case studies to develop training materials.

Case Studies

The project indicator monitoring framework The project indicator working sheet The main activities The results (Hebei Expressway) Some issues and findings in the case

studies.

Thank you! 谢谢！