Monitoring & Evaluation TIPS & TRICKS

TTI PEC Anglophone Africa Regional Peer Learning Workshop Coconut Grove Regency Hotel | Accra, Ghana

Wednesday, July 16th, 2014 Dr. Mo Adefeso-Olateju and Rebecca Oluwatoyin Doherty

AGENDAIntroductions

Name, Organization, Brief Description of Project & Phase

Objectives

Reflective Activities

Introduction to monitoring & evaluation

Developing an M&E System

Practical challenges with M&E

Conducting Project Evaluations

Conclusion

OBJECTIVES!

!

To provide hands-on support in developing and utilizing M&E tools

The session will be focused on practical support and will be as interactive as possible

A brief overview presentation on previous M&E presentations and materials shared at TTI PEC events

Introduces potential tools and resources to assess outcomes

Explore how to feed results back into strategy &

REFLECTThe first time I heard M&E was _____________________________. !I t h o u g h t M & E w a s _____________________________. !I later discovered that M&E was _____________________________. !Now I use M&E in my work by _____________________________. !Today, I am interested in working on a/an (planned, ongoing, or completed) __________ project. !I did my baseline survey in/on _________________________________.

Designing & setting upthe M&E syste

Gathering managing informatio

Reflectincritically toimprove actio

Communicating reporting result

Key parts of a M&E system

M & E and Project Planning: “THE CHICKEN, OR THE EGG?”

SHOULD WE HAVE M&E PERFORMANCE INDICATORS BEFORE A WORK PLAN? SHOULD WE DEVELOP AN M&E PLAN BEFORE THE PROJECT PLAN?

WHAT DO WE DO WITH THE RESULTS OF OUR MONITORING?

M&E IN THOUGHT AND PRACTICETRUE or FALSE:

I always conduct a needs assessment before conceiving a project _________________. !

I should always conduct a needs assessment before conceiving a project _________________. !I start thinking about stakeholder participation at the onset of my project planning _________________. !My Monitoring/Evaluation/M&E is always participatory in process and results ___________________. !

We always reflect on the information that we collect for M&E _____________________. Please explain how. !

M&E seems to be more of a measurement tool/activity than a planning or evaluative activity.   !

Whilst measurement (through well-developed performance indicators) is critical to M&E, M&E is a much broader, organization-wide and learning process.  

M&E: A SNAPSHOT OF OUR GROUPPARTICIPANTS KNOW WANT TO KNOW LEARNED

- Gloria Eboh, African Heritage Institution (AfriHeritage) - Sola Oluwadare, African Heritage Institution (AfriHeritage)

I would like to do this by being able to analyze the data gotten from the M&E tools accurately

- Analyzing the M&E results; - How to improve on the M&E tools I am using currently.

Simone Fugar. Institute of Economic Affairs, Ghana

- Adopt better online monitoring tools; - Knowledge of the most efficient online monitoring tools

- Drusilla David, Centre for the Study of the Economies of Africa, Abuja, Nigeria.

- Anumudu Peace Onyinyechukwu, Centre for the Study of the Economies of Africa.

- CSEA uses the Impact Log as an M&E tool.

We will like to improve the log to enable us- capture in detail the activities of the centre- outcome of its research findings- improve on the impact log to enable it measure its impact to

its target audience, policymakers and stakeholders to determine if its messages are been used in the policymaking sphere

- to enhance development based outcomes in the long,medium or short term as the case may be.

- How to use other performance indicators to measure and capture specific communication activities adequately and regularly.

- How to measure outcomes from different information products and how their impact and influence stands.

- How to strengthen our monitoring and evaluation skills by incorporating other tools and methods

- as well as capacity building in tool use to ensure regular data capturing.

- Vicentia Kotia, - I use an adapted version of the impact log sheet, a simple excel sheet

- No topic in particular but I'd love to learn and get better at M&E - Hands on training on the use of simple but effective M&E tools

- Job Eronmhonsele, & Peter Ugholeh. Centre for Population and Environmental Development (CPED) Benin City, Nigeria

- I developed to track on-going activities and outputs Just a framework on the website for knowing the number of site visitors within a given period, - and a CHART (still undergoing development) to help monitor CPED invitation to participate and make presentations in conferences and workshops around the globe.

- We would like to have a robust M & E tool so as to track visitors and get feedback

- as well as know which of the publications appeal to visitors. - We would like to learn how to use Google Analytic; Hoot

suite; Twitter Counter

Food for ThoughtYour project plan should have been from the M&E perspective.

E.g. Consultations with the target population informs the needs assessment and the project as the most feasible/desirable/affordable solution.

NATURE OF THE PROJECT/BRIEF DESCRIPTION

M&E should also be a part of your work plan.

E.g. at this stage of the project, this is what we expect to see

M&E affects program design because of your means of collecting data

Needs Assessment; Assets Assessment/Capacity Assessment

Start with an M&E Plan. A good M&E Plan will help a good project…or at least, side-by-side

Let us re-do our work plan. Have we laid the foundation for a bungalow? And now trying to build a duplex?

Result is a change in behavior. Measure success by outcomes, and not outputs. We often celebrate the result of outputs, and not outcomes or impact. Output is what you put out; however, results should be measured by the success of people (Dr. Chuks Eresia-Eke)

SHORT VIDEO!https://www.youtube.com/watch?v=MUSzePBTpAY

Introduction to monitoring and evaluation

! Monitoring: ! the regular collection and analysis of

data/information ! communication to strengthen project

partnership. ! Evaluation: ! occasional assessment of the overall

value and progress of a project

Common definitions of M&E

Looking critically at M&EMonitoring = process of collecting and interpreting data to provide management and stakeholders with evidence of the extent of progress towards achieving the objectives from the implementation of a plan !!!Evaluation = process of determining whether a particular plan was well designed (was it relevant, did it have a significant impact and was it sustainable?) and well implemented (was it effectively and efficiently implemented?) → leads to lessons learned for decisions

Changing the M&E Perspective

! M&E is mainly for the external funding body

! Focus on activity & financial reporting

! Lots of data & little analysis ! Little learning ! Little stakeholder participation ! M&E is about highly qualified

technical experts conducting statistical studies

! M&E is boring and not very useful for project staff/beneficiaries

Shifting Paradigms, methodologies, approaches & tools

■ M&E is for everybody involved in the initiative

■ M&E helps in understanding impact & reasons for success/failure

■ Analysis & critical reflection to improve the initiative

■ Learning is a key function of M&E

■ People processes – lots of stakeholder participation

■ M&E is active, interesting and useful to all involved in an initiative

Developing an M&E System

What is an M&E system?• An M&E system is

• A set of planning, information gathering and synthesis, • reflection and reporting processes, • along with necessary supporting conditions and capacities required

for the M&E outputs to make a valuable contribution to project decision-making and learning.

Setting up a M&E system

! Practical Steps 1. Define the purpose and scope

2. Decide what to monitor & evaluate 3. Plan information gathering &

organisation

4. Plan critical reflection processes & events

5. Plan for quality communication & reporting

6. Plan for the necessary conditions & capacities

Key parts of a M&E system

Designing & setting upthe M&E system

Gathering andmanaging information

Reflectingcritically toimprove action

Communicating andreporting results

M&E as an integral part of project management

The project strategy!(what will be achieved and how it will be achieved)

Detailed operational plan

Project outputs, outcomes and

impacts

The M&E!system!!

!

!

! Con

tinua

l ref

inem

ent

Communicating and!reporting results

Reflecting critically!to improve action

Gathering and!managing information

Developing!the M&E system

Field data

informati

on

Basis for Basis for

Leads to …

Mutual

refinements

Improvements through M&E

Integrated Learning Processes in M&E - Illustration

jan marfeb apr sep novoct decmay julyjune aug

NGO review meeting

Project mgt coord comm meeting

Inter-agency steering committee meeting

Special review on extension activities

Quarterly report

Half-yearly & yearly report

WORKSHEET

Reflection processes & events - Example

WORKSHEET

Results framework

= framework used to assess performance ▪ Expectations ▪ Indicators ▪ Sources of information ▪ Baselines and targets + Reports for planning

The 3 Spheres (Tsui et al 2014)

Results chain

RESULTS

Inputs ! Activities ! Outputs ! Outcomes ! Impact

Short-term

Medium-term

Long-term

Resources Processes

Within control of the organization Partial Org. Control No Org. Control

Sphere of Control: Advocacy, Tactics, Inputs, Activities, Outputs

Sphere of Influence: Changes in behavior,

relationships, attitudes, agenda

Sphere of Concern: Policy

& Political Context

WORKSHEET

Results chain – Example

WORKSHEET

Practical challenges with monitoring and evaluation

Insufficient Consideration of M&E at Initial Project Stages

! Unintentionally, M&E is often set to fail during the project design ! No clear direction is given in the project design on how M&E system will be

developed. ! No adequate budget for M&E ! No sufficient time and expertise allocated to M&E

Failure to align Phases of M&E with Phases of the Project

WORKSHEET

Challenges in defining indicators

! More often indicators are stated and left at the level of a logical framework.

! No attempt is made to develop an indicator matrix that gives the details

! See example………………FAWE Uganda, Asante [Number with reference to logframe] [Name of indicator] Definition Precise definition [=Formula] What it measures [A narrative explanation of the importance of the indicator] Frequency [For example, monthly quarterly, annually, every two years etc] Disaggregation [For example, by grade, sex, school type, geographical area etc] Source Notes Responsibility [Who is responsible in the M&E Unit for collecting the information]

Developing Effective Indicators !

!Indicator = measure (of performance, progress, condition) !

Properties of good indicator: ▪ Specific = leaves no ambiguity ▪ Measurable = can be compared ▪ Attainable = a source readily exists ▪ Relevant = measures intended progress ▪ Time-bound = refers to specific time

Indicators, baseline and targets (& Planning) OUTCOME

ACTIVITY OUTPUT Short- to

medium-term Medium-term Medium- to long-term IMPACT

OBJE

CTIVE

Design training to increase community

involvement

Training of SMC members on community involvement

SMC members are actively involved in school management

A school supported by the community helps increase attendance

Increasing attendance will yield lower

repetition and higher completion

Higher completion will improve learning

INDI

CATO

R

Number of SMC guidebooks printed

Number of SMCs where at least 3

members have been trained

Percentage of SMCs which are actively involved in school

management Attendance rate Repetition rate

Percentage of pupils who master specified

competencies

BASE

LINE

O 10% 10% 75% 8% 30%

TARG

ET

5000 50% by year X 25% by year X 80% by year X 6% by year X 35% by year X

WORKSHEET

The importance of monitoring outcomes, and not only outputs –

Practical way of conducting evaluations

• Considerations

• Processes

Evaluation

! A systematic (and as objective as possible) examination of a planned, on-going or completed project.

! It aims to answer specific management questions and to judge the overall value of a project and supply lessons learned to improve future actions, planning & decision making.

(Some) Purposes of Evaluation

! Learning: Generating new insights/knowledge in order to contribute to improved action in the future

! Accountability: Ensuring that an initiative is answerable to its beneficiaries, partners & donors for effective and efficient delivery and for appropriate use of funds.

Different types of project/programme evaluations

! Mid term evaluations (Formative/process evaluations) – To provide feedback on the implementation process

! End of project evaluation (Summative) – To assess outcomes, impacts, sustainability, external utility & lessons learnt

! Self assessments – Internally guided & controlled. To gather information and provide reflection, discussion, debate & improvements

**The above could be internal/external – depending on who is commissioning and/or taking a lead in undertaking them

Key steps in a project evaluation

Source: IUCN 2004Source: IUCN 2004

Common evaluation questions

The Situation to Improve

Problems and Visions

Planned (Goal, Purpose,

Outputs and Activities)

Inputs Activities Actual Outputs

4. IMPACT

2. Efficiency

1. Relevance

3. Effectiveness

5. Sustainability

Evaluation questions

1. Relevance - Was/is the project a good idea given the situation to improve? Was the logic of the project correct? Why or Why Not?

2. Effectiveness - Have the planned results been achieved? Why or Why Not

3. Efficiency - Have resources been used in the best possible way? Why or Why Not?

4. Impact - To what extent has the project contributed towards its longer term goals? Why or Why Not? Have there been any unanticipated positive or negative consequences of the project? Why did they arise?

5. Sustainability - Will there be continued positive impacts as a result of the project once it has finished? Why or Why Not?

References & Resources!

• Barnett, C. and R. Gregorowski (2013). ‘Learning about Theories of Change for the Monitoring and Evaluation of Research Uptake.’ IDS Practice Paper In Brief 14. Publisher IDS. (http://www.ids.ac.uk/publication/learning-about-theories-of-change-for-the-monitoring-and-evaluation-of-research-uptake)

• Batchelor, S. (2012) Information Eco-systems of Policy Actors. https://www.ids.ac.uk/publication/information-ecosystems-of-policy-actors-reviewing-the-landscape

• Beynon, P., Chapoy, C., Gaarder, M. & E. Masset (2012). ‘What difference does a policy brief make?’ IDS, 3ie and NORAD. (http://www.3ieimpact.org/en/evaluation/policy-influence/policy/)

• Doemeland, D. & J. Trevino (2014). ‘Which World Bank reports are widely read?’ • Outcome Mapping: Learning Community http://www.outcomemapping.ca • Stachowiak, S. 2007. Pathways for Change: 6 Theories about How Policy Change Happens.

Organizational Research Services. • Sumner, A. et al. (2011). ‘What shapes research impact on policy? Understanding research

uptake in sexual and reproductive health policy processes in resource poor contexts.’ Health Research Policy & Systems. Vol. 9.

• Tsui et al (2014). ‘Monitoring and Evaluation of Policy Influence and Advocacy.’ ODI Working Paper.

Thank You!

Questions/Comments/Feedback

For further inquiries, please contact:

Dr. Modupe Adefeso-Olateju, mo.olateju@tepcentre.com Rebecca O. Doherty, doherty.rebecca@gmail.com

THANK YOU