Monitoring & Evaluation
IN PRACTICE
TTI PEC Anglophone Africa Regional Peer Learning Workshop
Coconut Grove Regency Hotel | Accra, Ghana
Wednesday, July 16th, 2014 !!Mo Adefeso-OlatejuRebecca Oluwatoyin Doherty
Linking Monitoring, Evaluation, & Measurement
SHORT VIDEOS
Importance of monitoring & evaluation
https://www.youtube.com/watch?v=2mBcb-AQ8xQ 4:00minutes - end !Indicators https://www.youtube.com/watch?v=SZqOKW2_jNU !Indicators: Basic Concepts https://www.youtube.com/watch?v=qmrDNuJzRv4
1ST THINGS FIRST
4 BASIC M&E POINTS MUST BE FOLLOWED
Who is our population? What data is to be collected from them? Or about them?
How will the data captured enhance communications? And also fit within the rest of the M&E analysis?
How will the data be analyzed? Who will be doing the analysis? What do we know about our baseline data?
How will the process be managed? And by whom?
An organizational assessment is needed…in order to establish a process for M&E!
Source: https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies
ASSESSMENT OF READINESSIn order for the M&E Function to succeed, it is necessary to meet two major conditions: !▪ Organizational readiness
– Is there a clear M&E framework with targets?
– Is the planning process evidence-based? – Is the TT focused on achieving results (leadership, communications) !▪ Technical readiness - Do the staff have the relevant experience and skills?
WORKSHEET
What do we monitor? Inputs. Activities. RESULTS.
WORKSHEET
Results chain
Resources Processes
Within control of the organization Partial Org. Control No Org. Control
Sphere of Control: Advocacy, Tactics, Inputs, Activities, Outputs
Sphere of Influence: Changes in behavior,
relationships, attitudes, agenda
Sphere of Concern: Policy
& Political Context
WORKSHEET
WORKSHEET
Results chain – ExampleWORKSHEET
Indicators, baseline and targets (& Planning)WORKSHEET
Strategies & approaches for identifying evidence of change
• Signposts
– output log, web analytics, media tracking, citation tracking
– discourse analysis, social network analysis
• Outcome (or influence) log
– record of interactions with decision-makers, anecdotes shared by stakeholders (e.g. with the director)
• Feedback mechanisms
– stakeholder surveys
– interviews
– focus group discussions
WORKSHEET
Integrated Learning Processes in M&E - Illustration
jan marfeb apr sep novoct decmay julyjune aug
Quarterly report
Half-yearly & yearly report
WORKSHEET
BREAK Questions, Answers, Comments
WEBSITES: www.raguide.org www.resultsaccountability.com
BOOK ORDERS: www.trafford.com www.amazon.com
Results Accountability Decision-makingand Budgeting
Fiscal Policy Studies Institute Santa Fe, New Mexico
“If you do what you always did,
you will get what you always got.”
Kenneth W. JenkinsPresident, Yonkers NY NAACP
TO THINK OR NOT TO THINK ABOUT M&E? FOOD FOR THOUGHT…
The first step in performance accountability is toDRAW A FENCE
Around something that has ORGANIZATIONAL OR FUNCTIONAL IDENTITY
The Whole Organization
Division ADivision B
Unit
Division C
FunctionUnit 1
WORKSHEET
HOW DO WE THINK ABOUT WHO, WHAT & WHY WE
MEASURE?
Different Kinds of Progress1. Data
a. Population indicators Actual turned curves: movement for the better away from the baseline.
b. Program performance measures: customer progress and better service: How much did we do? How well did we do it? Is anyone better off?
2. Accomplishments: Positive activities, not included above.
3. Stories behind the statistics that show how individuals are better off.
What’s Next?A Basic Action Plan for Results Accountability
TRACK 1: POPULATION ACCOUNTABILITY● Establish results ● Establish indicators, baselines and charts on the wall● Create an indicators report card● Set tables (action groups) to turn curves
TRACK 2: PERFORMANCE ACCOUNTABILITY● Performance measures, and charts on the wall for programs, agencies and service systems● Use 7 Questions supervisor by supervisor and program by program in management, budgeting and strategic planning
Input
Activity
OutputOutcome
Goal
Logic Model
WORKSHEET
If I can…. Measure, Track, Improve & Show Performance
Education Performance Accountability
What’s in it for me?
Then…. here’s what’s in it for me:
! Improvement = more money
! Success at the State’s Accountability Game
! Student and Parent Satisfaction
! Tool vs. fingers crossed
! Knowing where your students are between achievement tests
! Knowing that you’ve made a difference
WORKSHEET
WORKSHEET
WORKSHEET
Turn the Curve Exercise: Program Performance5 min: Starting Points - timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s)10 min: Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast – OK or not OK?15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works
10 min: Report Convert notes to one page
Two pointers to action
WORKSHEET
Program: _______________Performance Measure
(Lay definition)PerformanceMeasureBaseline
Story behind the baseline --------------------------- --------------------------- (List as many as needed)
Partners --------------------------- --------------------------- (List as many as needed)
Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost
ONE PAGE Turn the Curve Report: Performance
Sharp Edges
4. --------- Off the Wall
4. --------- Off the Wall
WORKSHEET
2 customers
1 measure
1 measure
Discuss how we’re doing
2 partners
2 what works ideas (1 nc/lc)
Discuss how to implement
20 MinuteExercise
40
WORKSHEET
1 population
1 result
1 indicator
Discuss how we’re doing
2 partners
2 what works ideas (1 nc/lc)
Discuss how to implement
20 MinuteExercise
1. What population are we concerned about? (e.g. U.S. population)
2. What condition(s) do we want for this population? (Americans are free of cancer.)
3. How could we measure these conditions? ( (e.g. cancer rates)
4. How are we doing on the most important of these measures? (baseline history)
5. Who are the partners with a role to play in doing better?6. What works – what would it take – to do better? (What is our role?)7. What do we propose to do?
POPULATION ACCOUNTABILITYWORKSHEET
Results Based Grantmaking ExerciseA. Result _________________________B. Indicators 1. _______________ 2. _______________ 3. _______________
For each: Getting Better Getting Worse or About the Same?
C. Story behind the baselines (Why better or worse?) __________ __________ __________ __________
D. Partners with a role to play __________ __________ __________ __________
E. What would it take? (5 most important strategies) 1. _____________ 4. ______________ 2. _____________ 5. ______________ 3. _____________F. Our Roles (3 best ideas) 1. __________________ 3. _________________ 2. __________________ Off the wall:. _________
10 min:
10 min:
10 min:
15 min:
10 min:
WORKSHEET
Performance Measurement: How to Develop Them Sectoral Examples
See Also: raguide.org Questions 3.10 and 3.11
● # of people served
● % participants who got jobs
● staff turnover rate
● # participants who got jobs
●% of children reading at grade level ● cost per unit of service
● # applications processed
● % patients who fully recover
What Kind of PERFORMANCE MEASURE?
Upper Left
Lower Right
Upper Right
Lower Left
Lower Right
Upper Left
Upper Left
Lower Right
QUANTITY !!
How much did we do?
QUALITY !!
How well did we do?
E F O R T
E F E C T
Is anyone better off?
WORKSHEET
How much did we do?
Professional Development
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rt
Customers # teachers served# students# parents Activities # training sessions# hours of training # study groups# TA contacts # mentoring relationships
Common measures% teachers certified in subjects they teach% customers satisfied with how service is delivered % staff with good morale Unit cost per training unit (HL #3) Activity specific measures% positive evaluations on training% attendance in study groups% of faculty participating in study groups % mentoring pairs who complete mo. rept. % mentees satisfied with match
# # !# # !# !# # #
% teachers satisfied that prof dev helped them be a better teacher (DDA #2) % teacher retention rate - total - “quality” teachers (DDA #1)% mentees who report relationship helped build their skills % teachers who demonstrate new skills - by observation (HL #2) - by self report % students in “trained” classrooms with improved test scores (HL #1)% students with reduced behavior problems % students with good school attendance% teachers who report better use of time
SAMPLE WORKSHEET
How much did we do?
Professional Development
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber of students &
teachers served !
Number of workshops
Unit cost !
% of teachers with PD plan
# teachers who used training (by self rept or observation) !# Teachers retained !# Achievement in classroom with trained teachers
% teachers who used training (by self rept or observation) Teacher retention rate !Achievement rates in classroom with trained teachers
How much did we do?
Environment: Water Quality
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber of
stream stations monitored
Average sitesper monitorper month
Number miles of healthy streams
Percent miles of healthy streams
How much did we do?
Commerce Tourism
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber of
books sent to interested
consumers
Cost perinquiry
delivered
Number oftourists
Number touristbusinesses
Tourism market share
Growth in tourist industry
How much did we do?
Personnel Department
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtNumber ofApplicantsProcessed
AverageRecruitment
Period
Workforce New Hires
Workforce Turnover Rate
(non-promotions)
CustomerSatisfaction
How much did we do?
Every Program(Financial Performance Measures)
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rt
Cost(total)
UnitCost
Benefit Value (total)
Cost-Benefit(ratio)
How much did we do?
Corrections
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rt # Inmates Rate ofovercrowding
# Recidivism
% Recidivism
How much did we do?
Blank
How well did we do it?
Is anyone better off?
Quantity Quality
E
ffect
Effo
rtX X
X X
PLANNING ALONG M&E
OBJECTIVES
PERFORMANCE
INDICATORS
INPUTSACTIVIT
IESOUTPUTS
OUTCOMES
IMPACT TIMELINERESPONSIBILITY
WORKSHEET
More ResourcesONLINE RESOURCES http://resultsaccountability.com/ !http://researchtoaction.org !http://www.dgmt-growingconfidence.co.za/ !YOUTUBE TRAINING VIDEOS !https://www.youtube.com/watch?v=MUSzePBTpAY !https://www.youtube.com/watch?v=2mBcb-AQ8xQ !https://www.youtube.com/watch?v=SZqOKW2_jNU !https://www.youtube.com/watch?v=qmrDNuJzRv4
More Resources M&E Online Tools
!ALT METRICS http://www.researchtoaction.org/2013/09/how-to-harness-the-power-of-altmetrics-a-researchers-guide/ http://www.researchtoaction.org/2013/08/altmetrics-and-the-global-south-increasing-research-visibility/ !USING GOOGLE TO TRACK AND IMPROVE YOUR RESEARCH IMPACT http://www.researchtoaction.org/2013/08/using-google-to-track-and-improve-your-research-impact/ http://opendocs.ids.ac.uk/opendocs/handle/123456789/2878#.U8VcXl7yDwJ !GOOGLE ANALYTICS http://www.researchtoaction.org/2013/11/key-indicators-for-google-analytics/ http://www.google.com/analytics/ !HOW TO GET STARTED WITH GOOGLE ANALYTICS http://mashable.com/2011/05/23/how-to-use-google-analytics/ !
M&E in Theory and Action
!MONITORING & EVALUATION: SOME TOOLS, METHODS & APPROACHES http://gametlibrary.worldbank.org/FILES/215_M&E%20Guidelines%20on%20Indicators%20and%20Data%20Sources.pdf !THEORY OF CHANGE VS. LOG FRAMES http://www.researchtoaction.org/2012/05/theory-of-change-vs-logframes/
http://www.knowledgebrokersforum.org/wiki/item/theory-of-change-vs-logframes
http://mande4mfi.wordpress.com/category/theory-of-change/
!CASE STUDIES ON M&E https://sites.google.com/site/newmediacommunications2010/mobiles-and-m-e/2-case-studies !EXAMPLES OF PROGRAM OUTCOMES www.whatworks.org
EVALUATION PLANS
http://nonprofitanswerguide.org/faq/evaluation/what-is-an-evaluation-plan-and-why-create-one/
Questions/Comments/Feedback
For further inquiries, please contact:
Dr. Modupe Adefeso-Olateju, [email protected] Rebecca O. Doherty, [email protected]
THANK YOU
Top Related