INTERNATIONAL FINANCE CORPORATION Workshop on IMPACT ASSESSMNENT AND RESULTS EVALUATION: Developed...

INTERNATIONAL FINANCE CORPORATIONINTERNATIONAL FINANCE CORPORATION

Workshop on IMPACT ASSESSMNENT AND

RESULTS EVALUATION:

Developed by Alexander Kobzev

May 24, 2003

IFC KIEV

Introduction: anticipated results of this training

Upon completion of the training Ukraine Agribusiness Development Project:

UADP team members would know how IFC monitoring and evaluation system is workingAppropriate team members would fill in monitoring tool correctly and use informationInput of UADP team members will be incorporated in monitoring tool

Any other expectations?Any other expectations?

TA MONITORING AND EVALUATION WAS DESIGNED TO MEET TWO KEY OBJECTIVES:

To evaluate project progress and learn from best practices

To demonstrate projects transparency and to promote results to PEP’s donors and IFC worldwide

Project Design

Clearly define goals and objectives

(Keep in mind how you will measure their achievement )

SOM, IAM

Suggest our approach or meet specific donor requirements

SOM

Agree measurable results of the project (through M&E) framework

- Verify goals and objectives- Define actions, timing- Set targets

SOM, IAM, PM and key team members (if appropriate)

Baseline data collection at the very early stage

Project team or external source

Timely collect progress data and adjust project plan (if appropriate)

Appointed team member under guidance of PM and SOM

Critical steps

Responsibility

FundraisingProject

PlanningAction

Monitoring &Evaluation

REQUIRED STEPS AT VARIOUS STAGES OF PROJECT LIFE

FRAMEWORK FOR RESULTS MONITORING AND EVALUATION

Goal Objectives Activities Output Outcome Impact

What we are aimingto change

To increase financing to SME through the development of leasing as a financial instrument

What should be implementedto achieve the desired change

To improve/ increase leasing service

To improve knowledge and attitude of potential customers of leasing service

To provide legal assistance...

Our physical actions

To deliver 24 trainings, 2 booklets published, 5000 copies each

-Direct results from the Activities- Have effect immediately or in a short term period during the project timeframe

600 trained employees of financial institutions

75% of participants had significantly improved understanding of leasing transactions(measured by test)

- Anticipated effect - Can be observed during the project life and within 1-3 years after completion- Measures the achievement of Objectives

-Increased number of financial institutions providing leasing service -At least 4 as a direct outcome of the project

- Desired final change - Has effect within 1-10 years after completion, rarely during the project timeframe- Measures the achievement of Goal

Leasing companies growing their business

More leasing finance is available to SMEs

Definition

Example

FIRST TOOL USED TO STRUCTURE PROJECTS AND TO SET TARGETS E.g. Technical Assistance for Sugd Agro Serv

ACTIVITIES: ACTIVITIES TARGETS:EXPECTED OUTPUTS:

OUTPUT TARGETS:EXPECTED

OUTCOMES:OUTCOME TARGETS: EXPECTED IMPACT: IMPACT INDICATORS:

Priority A

Deliver trainings/ consultations to SAS shareholders and farmers in the broader Sugd province

24 sessions of the basic course (basic course =60 hours) and 8 sessions of advanced course (advanced course =100 hours)a) crop productionb) sound environmental practices, including irrigation managementc) occupational health and safety

At least 400 participants have received basic training and at least 100 - advanced training in:a) crop productionb) sound environmental AG practicesc) occupational health and safety

80% of participants have passed test w ith a pass mark of 75%

On all SAS shareholder farms w ater table is low er than 2 meters at all locations by 2007

SAS area of salinized land does not increase

Increased SAS area under active use of biological control systems

Full adoption of integrated pest management by SAS

Set up and maintain demonstration f ield

Establish 25 hectares demo site, involving other supplier companies in the region and governmental agencies

Provide f ield training and advice to SAS members

8 f ield days a year, (20 in total) covering a) laser levellingb) irrigationc) seedingd) tillagee) harvest management

Model is replicated in other Tajik and CIS areas # of replications

SAS shareholders and farmers in the broader Sugd province improve their know ledge of modern crop production practices

Improvement in:1.1 Salinity management1.2 Pest management1.3 Fertilizers use1.4 Irrigation1.5 Crop diversif ication1.6 Safety of farming

At least 40 participants per f ield day (~800 in total)a) laser levellingb) irrigationc) seedingd) tillagee) harvest management

Objective 1: To introduce modern agronomic practices

What are we actually doing? What are the immediate results of our action? What should be implemented?

All SAS hectares receive accurate timimg, rate and placement of fertilizers by 2006

30% reduction in w ater use per SAS hectare by 2006

Flexibility in crop production is resulting in profitable operations each year

Decreased accident and mortality rate

Poverty is reduced through:

Crop productivity gradual increase

Farms earn more income and become more profitable

Farmers debt burden is servicable

Farmers earn more for their families

Crop yield per hectare increases by 50% by the year 2010

Average net income increases to 100USD per hectare

Loan defaults less tha 3%Debt per hectare is servisable

Income per farm w orker increased to 20 USD per month by 2007

Where it should bring us at the very end?

SECOND TABLE USED AS A PLANNING TOOL FOR MONITORINGE.g. Central Asia Leasing Project

II 2002 III 2002 IV 2002 I 2003 II 2003 III 2003

Develop and deliver customized training in each country to government off icials and businesses

# of customized training modules# of seminars for government off icials# of seminars for f inancial institutions# of seminars for lessers accountants# of seminars for lessees

Internal reports Quarterly Team assistasnt 9-12 (total)40502040

42011

Provide individual consultation to lessers, lessees, potential lessees and lessers, local commercial banks, microfinance institutions

# of hours of consultation provided# of consultations provided

Internal reports Quarterly Team assistasnt 1200

1000

55

II 2002 III 2002 IV 2002 I 2003 II 2003 III 2003# of trained participants:~300 governmental off icials~700 fin institutions rep's~180 lessers' accountants~1000 SME representatives

Internal reports Quarterly Team assistasnt3007001801000

3004020

# of organizations served Internal reports Quarterly Team assistasnt 600(200 per country)

16

II 2002 III 2002 IV 2002 I 2003 II 2003 III 2003Governmental off icials are more supportive to leasing

Local leasing institutions report to have support from local off icials

Leasing institutions survey

Annually Deputy project manager

N/a N/a

Financial institutions launch leasing or improve leasing operations

# of participating companies launching leasing operations# of participating companies improving leasing operations



4

6

N/a

N/a

Lessers take all available tax exemptions

% of participating companies improve profitability



90% N/a

Private sector is more active in attaining f inance through leasing

% of participating SME consider choice betw een credit and leasing% of participating SME apply for leasing

SME survey (comparison of participating and other comppanies)


40%

10-20%

N/a

RESPONSIBILITYFREQUENCY

RESPONSIBILITY

TARGETACTIVITIES: OUTPUT INDICATORsDATA SOURCE &

COLLECTION METHODs

Objective 2: To improve/increase the knowledge and understanding of equipment leasing

TARGETCURRENT STATUS

OUTCOME OUTCOME INDICATORsDATA SOURCE &

COLLECTION METHODs

FREQUENCY

CURRENT STATUS

OUTPUTS:

DATA

DATA

DATA

Potential lessees and lessers, and local off icials understand concept and practicalities of leasing

OUTPUT INDICATORsDATA SOURCE &

COLLECTION METHODs

FREQUENCY RESPONSIBILITYCURRENT STATUS

TARGET

PROJECTS BECOME MORE EFFECTIVE

Project will train fourteen local Uzbek consultants over two separate rounds with each round lasting some three weeks

The project team decided to allow consultants a period of six months to implement their training

In March 2002 an evaluation of the first round was conducted through personal interviews with consultants and their clients

Result: redesign of program for 2nd round

Result: redesign of program for 2nd round

Key findings:– Consultants showed improved knowledge of key business specializations

where they previously had little or no exposure– During the six given months they progressed toward service

diversification, project/client management, but less than expected

Eg.: Pilot Training of Uzbek Consultants - Midpoint Evaluation

Results for Activity 1Results for Activity 1

Activity Specification:

Produce informational and analytical materials;

Carry out consultations on legislative issues;

Participate in working group activities.

Activities Outputs

Issues addressed in produced policy papers(concepts, proposals), the extent of papers distribution;

Different groups/ministerial departments consulted and issues addressed;

Submitted proposals and comments to working groups.

Outcomes

Policymaking process is facilitated;

Policymakers are better informed about agribusiness issues;

New strategies and legislation are proposed and developed by policymakers.

Outcome Verifiable Indicators

Provided advice is used by government;

New legislation specifically incorporates our suggestions;

Requests from government for information/advice or to participate in working groups.

Impact

Policy environment is more favorable for agribusiness development (needed legislation is adopted)

Activity:

Conduct and disseminate policy-oriented research and carry out consultations with policymakers



Provide consultations to banks, insurance companies and other financial intermediaries;

Promote interaction/be an intermediary between farms/agribusinesses and financial institutions

Organize and conduct training programs.

Activities Outputs:

Number of clients served and issued addressed;

Number of different companies consulted;

Number of contacts between farmers and financers created;

New concepts developed and presented;

Number of business plans developed and submitted to banks.

Outcomes:

Better access to credit, insurance secured;

Financial institutions more knowledgeable about agribusiness (and vice versa);

New financial products are introduced, including long-term credit.

Outcome Verifiable Indicators:

Expanded the number of banks, financial institutions providing credit, insurance;

Amount of credit, insurance provided and extended amount of long-term credit;

Increase the rate of repayment;

Percentage of crops insured;

Number of new financial products introduced.

Impact:

Farms become more profitable and efficient;

Stable market for long-term rural credit and insurance is established.

Activity:

Promote credit and insurance relations in the agricultural sector

Results for Activity 3: Agronomic ActivitiesResults for Activity 3: Agronomic Activities


Provide consultations, organize seminars on new crop varieties and use and management of agricultural inputs;

Set up and maintain demonstration fields introducing new crop varieties;

Be an intermediary between input-supplier and producers.

Activities Outputs:

Farmers served through carried out consultations and seminars;

Increased the number of farmers introduced to new crops and new technologies;

Training materials produced and extent of their distribution to ag producers;

Increased the access of input suppliers to ag producers and vice versa.

Outcomes:

New, more productive crop varieties planted;

New techniques for planting , input management adopted;

Better utilization of agricultural inputs;

Farmers are better educated on agronomic practices and better decision makers.


Number of new crops farmers cultivate;

Number of new inputs used; more economical/productive use of existing inputs

Extent of new technologies farmers use in their use.

Impact:

Increase in yields, crop productivity and in farm income increases

Activity:

Introduce new agronomic practices.



Publish bimonthly bulletin and other publications;

Organize and conduct press events (press clubs; meetings with the media) and prepare press releases;

Develop and maintain contacts with government, the media, NGOs, other research institutions, etc.

Activities Outputs:

Publication produced, issues presented to the public, and number of subscribers;

Number of media representatives and experts participated in events;

Number of interviews conducted and number of different media outlets that cooperate with the Project;

Number of contacts established and joint events conducted.

Outcomes:

Established project as a reliable source of information;

Provided regular and free access to information;

Facilitated contact among government, NGOs, media, researchers.


Number of times articles are referenced from Project’s publications;

Number of new subscribers, rate of re-subscription;

Number of articles, broadcasts about event.

Number of new requests for interviews

Impact:

Increase awareness of private agribusiness issues and gain support for private agribusiness.

Activity:

Conduct an information campaign through media events, publications, and public discussions.

OUTPUTS

CompletedActivities

Outputs

Conduct training on“Effective Presentation”

Provide 3 consultations toPolicy makers

Carry out study of ForestsSector in NW Russia

E.g. Nine UIP consultants have improved their presentation skills

Two draft amendments to Land Code were proposed

At least 15 potential investors receive the study report

Completed activities are not Outputs

- Direct results from the Activities- Have effect immediately or in a short term period during the project timeframe

OUTCOMES

- Anticipated effect - Can be observed during the project scope and within 1-3 years after completion- Measure the achievement of Objectives

Outputs vs Outcomes – What’s the difference?

E.g. How many people have improved their knowledge?

300 SME representatives were trained on advantages of leasing

How well they have applied the received knowledge

50 SME participated in the training haveapplied for lease financing

Outputs Outcomes

IMPACT

- Desired final change - Has effect within 1-10 years after completion, rarely during the project timeframe- Measures the achievement of Goal

Outcomes vs Impact – What’s the difference?

E.g. What was implemented?

Accurate fertilizingCrop management

How the implemented changes have affected the business?

Increase of yields per hectare, Revenue and profits

Outcomes

Impact

TARGET

- Planned level of the indicator

BASELINE

- Level of the indicator at the project starting point

WE SET TARGETS AND DEFINE BASELINE LEVEL TO HAVE CLEAR EXPECTATIONS FOR THE PROGRESS

HOW TO WORK WITH THE TABLE?

What should change?

Write short verbal description of the expected result in the first column

How you will measure?What’s your target for it?

Choose indicator and set target level for it

EXPECTED OUTPUTS:OUTPUT

TARGETS:

Players on leasing market of NW Russia improve their know ledge and skills of application and advantages of lease

100-150 managers from at least 50 companies trained or consulted

… …

COMPONENTS OF THE PROJECT Training

Objectives

ActivitiesOur means oftraining

OutputsHow many andhow well

peoplewere trained

OutcomesResults ofapplication of receivedknowledge(NB measureachievement

of Objective)

Potential indicators

- To introduce modern sustainable agricultural technologies in XX oblast

- To raise local management skills in financial controlling

- # of trainings delivered by subject- # of training hours- # of copies and key components of manuals

distributed- Name key components of online training tools

- # of people trained from # of companies- % of participants completed test with a XX

pass mark

- Increase in # of hectares under biological control systems

- Improve in profitability of participating companies

- % of participating SMEs apply for leasing- % of served companies improve internal

documents and CG practices

Comment

- Better avoid: To raise local management skills – too general objective, it’s better to be specific on the on the subject of training

- Better avoid: # of training modules and other training materials developed – additional measure, successfully conducted training assumes developed training materials

- Better avoid: Independent testing of the training materials – additional costs & questionable value

COMPONENTS OF THE PROJECT ‘Train the Trainer’

Objectives

ActivitiesOur means ofknowledgetransfer

Outputs

Outcomes How many

people and how wellare trained by the partners


- To raise local management skills in financial controlling trough building local training capacity

- To increase knowledge about CG of the next generation of managers

- # of seminars/round tables other events for partners

- # of trained partners - # of case studies, manuals, training modules

transferred

- # of partners delivering training over time- # of SME representatives trained by the

partners over time- % of participants completed test with a XX

pass mark

Comment

- Be specific on the on the subject of training

- Better avoid: Number of issued covered

- NB to identify target audience: e.g. business people vs students

- For internal information can be worthy to revisit partners and check quality of teaching

COMPONENTS OF THE PROJECT Legislative Work

Objectives

ActivitiesOur means ofinfluencing

thePolicy makers

OutputsWhat we propose

OutcomesWhat was adopted


- To improve legislative framework of XX sector

- # of meetings/round tables/other events held- # of presentation given during the - # of hours of consultation to government

- # of drafts/amendments proposed- Name of targeted laws to draft or amend

- # and names of approved drafts/amendments

Comment

Concern for further usability:- # of issues identified or discussed - # of collaborative activities and joint

activities- # of different departments consulted- # of other organizations requested our policy

input

COMPONENTS OF THE PROJECT Direct Assistance to Companies – Objectives and Scope Can Vary Greatly

Objectives

Activities

Outputs

Outcomes


- To support investment initiatives in particular sector

- To built up supply chain links for XXX company

- To increase access to XXX market of particular companies

- # of feasibility studies/business plans- # of potential partners evaluated- # of hours of consultation- # of deals consulted

- # of companies assisted with the tailored program

- # of partnerships created

- Value of investment in progress and completed in # of companies

- Additional sales, market share gain- Reduced costs, additional profits

Comment

Concern for further usability:- # of obstacles identified- Level of satisfaction of investors/clients- Independent valuation of research quality

COMPONENTS OF THE PROJECT Public Relations

Objectives

ActivitiesPR means

OutputsExpected

publicity

OutcomesPublicity reach and effect


- To increase public awareness of equipment leasing issues

- # of round tables/conferences/ press conferences held

- # of press releases sent- Deadline for website launch and frequency of

update

- # of media appearances per month/quarter and in total

- # of people reached from target audience per month/quarter and in total

- # of project replications

Comment

- PR is useful to to get more ‘clients’/partners or replicate the project. If there is no need in it then, we should consider usefulness of nation-wide campaigns

Concern for further usability:- # of new contacts established- #of requests from media to organize press

events- # of times articles/press releases are

referenced- # of new requests for interview

INTERNATIONAL FINANCE CORPORATION Workshop on IMPACT ASSESSMNENT AND RESULTS EVALUATION: Developed...

Documents

Transcript of INTERNATIONAL FINANCE CORPORATION Workshop on IMPACT ASSESSMNENT AND RESULTS EVALUATION: Developed...