Monitoring & Evaluation

IN PRACTICE

TTI PEC Anglophone Africa Regional Peer Learning Workshop

Coconut Grove Regency Hotel | Accra, Ghana

Wednesday, July 16th, 2014 !!Mo Adefeso-OlatejuRebecca Oluwatoyin Doherty

Linking Monitoring, Evaluation, & Measurement

SHORT VIDEOS

Importance of monitoring & evaluation

https://www.youtube.com/watch?v=2mBcb-AQ8xQ 4:00minutes - end !Indicators https://www.youtube.com/watch?v=SZqOKW2_jNU !Indicators: Basic Concepts https://www.youtube.com/watch?v=qmrDNuJzRv4

1ST THINGS FIRST

4 BASIC M&E POINTS MUST BE FOLLOWED

Who is our population? What data is to be collected from them? Or about them?

How will the data captured enhance communications? And also fit within the rest of the M&E analysis?

How will the data be analyzed? Who will be doing the analysis? What do we know about our baseline data?

How will the process be managed? And by whom?

An organizational assessment is needed…in order to establish a process for M&E!

Source: https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies

ASSESSMENT OF READINESSIn order for the M&E Function to succeed, it is necessary to meet two major conditions: !▪ Organizational readiness

– Is there a clear M&E framework with targets?

– Is the planning process evidence-based? – Is the TT focused on achieving results (leadership, communications) !▪ Technical readiness - Do the staff have the relevant experience and skills?

WORKSHEET

What do we monitor? Inputs. Activities. RESULTS.

WORKSHEET

Results chain

Resources Processes

Within control of the organization Partial Org. Control No Org. Control

Sphere of Control: Advocacy, Tactics, Inputs, Activities, Outputs

Sphere of Influence: Changes in behavior,

relationships, attitudes, agenda

Sphere of Concern: Policy

& Political Context

WORKSHEET

WORKSHEET

Results chain – ExampleWORKSHEET

Indicators, baseline and targets (& Planning)WORKSHEET

Strategies & approaches for identifying evidence of change

• Signposts

– output log, web analytics, media tracking, citation tracking

– discourse analysis, social network analysis

• Outcome (or influence) log

– record of interactions with decision-makers, anecdotes shared by stakeholders (e.g. with the director)

• Feedback mechanisms

– stakeholder surveys

– interviews

– focus group discussions

WORKSHEET

Integrated Learning Processes in M&E - Illustration

jan marfeb apr sep novoct decmay julyjune aug

Quarterly report

Half-yearly & yearly report

WORKSHEET

BREAK Questions, Answers, Comments

WEBSITES: www.raguide.org www.resultsaccountability.com

BOOK ORDERS: www.trafford.com www.amazon.com

Results Accountability Decision-makingand Budgeting

Fiscal Policy Studies Institute Santa Fe, New Mexico

“If you do what you always did,

you will get what you always got.”

Kenneth W. JenkinsPresident, Yonkers NY NAACP

TO THINK OR NOT TO THINK ABOUT M&E? FOOD FOR THOUGHT…

The first step in performance accountability is toDRAW A FENCE

Around something that has ORGANIZATIONAL OR FUNCTIONAL IDENTITY

The Whole Organization

Division ADivision B

Unit

Division C

FunctionUnit 1

WORKSHEET

HOW DO WE THINK ABOUT WHO, WHAT & WHY WE

MEASURE?

Different Kinds of Progress1. Data

a. Population indicators Actual turned curves: movement for the better away from the baseline.

b. Program performance measures: customer progress and better service: How much did we do? How well did we do it? Is anyone better off?

2. Accomplishments: Positive activities, not included above.

3. Stories behind the statistics that show how individuals are better off.

What’s Next?A Basic Action Plan for Results Accountability

TRACK 1: POPULATION ACCOUNTABILITY● Establish results ● Establish indicators, baselines and charts on the wall● Create an indicators report card● Set tables (action groups) to turn curves

TRACK 2: PERFORMANCE ACCOUNTABILITY● Performance measures, and charts on the wall for programs, agencies and service systems● Use 7 Questions supervisor by supervisor and program by program in management, budgeting and strategic planning

Input

Activity

OutputOutcome

Goal

Logic Model

WORKSHEET

If I can…. Measure, Track, Improve & Show Performance

Education Performance Accountability

What’s in it for me?

Then…. here’s what’s in it for me:

! Improvement = more money

! Success at the State’s Accountability Game

! Student and Parent Satisfaction

! Tool vs. fingers crossed

! Knowing where your students are between achievement tests

! Knowing that you’ve made a difference

WORKSHEET

WORKSHEET

WORKSHEET

Turn the Curve Exercise: Program Performance5 min: Starting Points - timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s)10 min: Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast – OK or not OK?15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works

10 min: Report Convert notes to one page

Two pointers to action

WORKSHEET

Program: _______________Performance Measure

(Lay definition)PerformanceMeasureBaseline

Story behind the baseline --------------------------- --------------------------- (List as many as needed)

Partners --------------------------- --------------------------- (List as many as needed)

Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost

ONE PAGE Turn the Curve Report: Performance

Sharp Edges

4. --------- Off the Wall

4. --------- Off the Wall

WORKSHEET

2 customers

1 measure

1 measure

Discuss how we’re doing

2 partners

2 what works ideas (1 nc/lc)

Discuss how to implement

20 MinuteExercise

40

WORKSHEET

1 population

1 result

1 indicator

Discuss how we’re doing

2 partners

2 what works ideas (1 nc/lc)

Discuss how to implement

20 MinuteExercise

1. What population are we concerned about? (e.g. U.S. population)

2. What condition(s) do we want for this population? (Americans are free of cancer.)

3. How could we measure these conditions? ( (e.g. cancer rates)

4. How are we doing on the most important of these measures? (baseline history)

5. Who are the partners with a role to play in doing better?6. What works – what would it take – to do better? (What is our role?)7. What do we propose to do?

POPULATION ACCOUNTABILITYWORKSHEET

Results Based Grantmaking ExerciseA. Result _________________________B. Indicators 1. _______________ 2. _______________ 3. _______________

For each: Getting Better Getting Worse or About the Same?

C. Story behind the baselines (Why better or worse?) __________ __________ __________ __________

D. Partners with a role to play __________ __________ __________ __________

E. What would it take? (5 most important strategies) 1. _____________ 4. ______________ 2. _____________ 5. ______________ 3. _____________F. Our Roles (3 best ideas) 1. __________________ 3. _________________ 2. __________________ Off the wall:. _________

10 min:

10 min:

10 min:

15 min:

10 min:

WORKSHEET

Performance Measurement: How to Develop Them Sectoral Examples

See Also: raguide.org Questions 3.10 and 3.11

● # of people served

● % participants who got jobs

● staff turnover rate

● # participants who got jobs

●% of children reading at grade level ● cost per unit of service

● # applications processed

● % patients who fully recover

What Kind of PERFORMANCE MEASURE?

Upper Left

Lower Right

Upper Right

Lower Left

Lower Right

Upper Left

Upper Left

Lower Right

QUANTITY !!

How much did we do?

QUALITY !!

How well did we do?

E F O R T

E F E C T

Is anyone better off?

WORKSHEET

How much did we do?

Professional Development

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rt

Customers # teachers served# students# parents Activities # training sessions# hours of training # study groups# TA contacts # mentoring relationships

Common measures% teachers certified in subjects they teach% customers satisfied with how service is delivered % staff with good morale Unit cost per training unit (HL #3) Activity specific measures% positive evaluations on training% attendance in study groups% of faculty participating in study groups % mentoring pairs who complete mo. rept. % mentees satisfied with match

# # !# # !# !# # #

% teachers satisfied that prof dev helped them be a better teacher (DDA #2) % teacher retention rate - total - “quality” teachers (DDA #1)% mentees who report relationship helped build their skills % teachers who demonstrate new skills - by observation (HL #2) - by self report % students in “trained” classrooms with improved test scores (HL #1)% students with reduced behavior problems % students with good school attendance% teachers who report better use of time

SAMPLE WORKSHEET

How much did we do?

Professional Development

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rtNumber of students &

teachers served !

Number of workshops

Unit cost !

% of teachers with PD plan

# teachers who used training (by self rept or observation) !# Teachers retained !# Achievement in classroom with trained teachers

% teachers who used training (by self rept or observation) Teacher retention rate !Achievement rates in classroom with trained teachers

How much did we do?

Environment: Water Quality

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rtNumber of

stream stations monitored

Average sitesper monitorper month

Number miles of healthy streams

Percent miles of healthy streams

How much did we do?

Commerce Tourism

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rtNumber of

books sent to interested

consumers

Cost perinquiry

delivered

Number oftourists

Number touristbusinesses

Tourism market share

Growth in tourist industry

How much did we do?

Personnel Department

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rtNumber ofApplicantsProcessed

AverageRecruitment

Period

Workforce New Hires

Workforce Turnover Rate

(non-promotions)

CustomerSatisfaction

How much did we do?

Every Program(Financial Performance Measures)

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rt

Cost(total)

UnitCost

Benefit Value (total)

Cost-Benefit(ratio)

How much did we do?

Corrections

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rt # Inmates Rate ofovercrowding

# Recidivism

% Recidivism

How much did we do?

Blank

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

Effo

rtX X

X X

PLANNING ALONG M&E

OBJECTIVES

PERFORMANCE

INDICATORS

INPUTSACTIVIT

IESOUTPUTS

OUTCOMES

IMPACT TIMELINERESPONSIBILITY

WORKSHEET

More ResourcesONLINE RESOURCES http://resultsaccountability.com/ !http://researchtoaction.org !http://www.dgmt-growingconfidence.co.za/ !YOUTUBE TRAINING VIDEOS !https://www.youtube.com/watch?v=MUSzePBTpAY !https://www.youtube.com/watch?v=2mBcb-AQ8xQ !https://www.youtube.com/watch?v=SZqOKW2_jNU !https://www.youtube.com/watch?v=qmrDNuJzRv4

More Resources M&E Online Tools

!ALT METRICS http://www.researchtoaction.org/2013/09/how-to-harness-the-power-of-altmetrics-a-researchers-guide/ http://www.researchtoaction.org/2013/08/altmetrics-and-the-global-south-increasing-research-visibility/ !USING GOOGLE TO TRACK AND IMPROVE YOUR RESEARCH IMPACT http://www.researchtoaction.org/2013/08/using-google-to-track-and-improve-your-research-impact/ http://opendocs.ids.ac.uk/opendocs/handle/123456789/2878#.U8VcXl7yDwJ !GOOGLE ANALYTICS http://www.researchtoaction.org/2013/11/key-indicators-for-google-analytics/ http://www.google.com/analytics/ !HOW TO GET STARTED WITH GOOGLE ANALYTICS http://mashable.com/2011/05/23/how-to-use-google-analytics/ !

M&E in Theory and Action

!MONITORING & EVALUATION: SOME TOOLS, METHODS & APPROACHES http://gametlibrary.worldbank.org/FILES/215_M&E%20Guidelines%20on%20Indicators%20and%20Data%20Sources.pdf !THEORY OF CHANGE VS. LOG FRAMES http://www.researchtoaction.org/2012/05/theory-of-change-vs-logframes/

http://www.knowledgebrokersforum.org/wiki/item/theory-of-change-vs-logframes

http://mande4mfi.wordpress.com/category/theory-of-change/

!CASE STUDIES ON M&E https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies !EXAMPLES OF PROGRAM OUTCOMES www.whatworks.org

EVALUATION PLANS

http://nonprofitanswerguide.org/faq/evaluation/what-is-an-evaluation-plan-and-why-create-one/

Questions/Comments/Feedback

For further inquiries, please contact:

Dr. Modupe Adefeso-Olateju, mo.olateju@tepcentre.com Rebecca O. Doherty, doherty.rebecca@gmail.com

THANK YOU