International engagement: m & e meeting
37
International engagement: m & e meeting Monitoring & Evaluation: an introduction for practitioners Liz Allen
description
International engagement: m & e meeting. Monitoring & Evaluation: an introduction for practitioners. Liz Allen. monitoring & evaluation can play a vital role in the management & improvement of an activity, organisation or process. Source: Subsetum ; http://www.flickr.com/photos/subsetsum/. - PowerPoint PPT Presentation
Transcript of International engagement: m & e meeting
International engagement: m & e meeting
Monitoring & Evaluation: an introduction for practitioners
Liz Allen
Source: Subsetum; http://www.flickr.com/photos/subsetsum/
monitoring & evaluation can play a monitoring & evaluation can play a vital role in the management & vital role in the management & improvement of an activity, improvement of an activity, organisation or process organisation or process
Monitoring & evaluation: why do it?
Theory & practice
M & E: challenges
Keeping it real: practical & proportionality
my talkMonitoring & evaluation: an introduction for practitioners
o accountability & validation
o management of funding initiatives
o strategy & planning
o policy & advocacy
o learning
my funder requires it …
I want my programme to
work well …
I want to do more in the
future …
I want to tell people about what we do … & show the benefits …
I want to find out what works …
rationaleMonitoring & evaluation: rationale
Theory
Source: Subsetum; http://www.flickr.com/photos/subsetsum/
o M & E frameworko Monitoringo Set up reviewo Process reviewo Formative reviewo Summative review
Source: Subsetum; http://www.flickr.com/photos/subsetsum/
o M & E frameworko Monitoringo Set up reviewo Process reviewo Formative reviewo Summative review o Inputs/activities
o outputso outcomeso … impact?o (longer term impact)
Source: Subsetum; http://www.flickr.com/photos/subsetsum/
o M & E frameworko Monitoringo Set up reviewo Process reviewo Formative reviewo Summative review o inputs/activities
o outputso outcomeso … impact?
(longer term impact)
Things to do & things to ‘measure/track’
Theory & formal frameworks
o Logical framework
o Outcome mapping
o Payback framework
o Theories of change
o Results-based management
Monitoring & evaluation: rationale M & E in project cycle
Monitoring & review pointsINPUTS & OUTPUTS
‘ M & E Framework’ design
Set-up/process review
INPUTS
Project start
Taking stockOUTPUTS/OUTCOMES
Learning & lessons for next time
Frameworks good for
o keeping outcomes in mind
o ‘forces’ definition of: o project objectiveso indicatorso appropriate data
collection method & timings
o organisationo simplification
Monitoring & evaluation: rationale
Frameworks less good
o keeping outcomes in mind
o ‘forces’ definition of: o project objectiveso indicatorso appropriate data
collection method & timings
o organisationo simplification
Monitoring & evaluation: rationale
o accommodating externalities & context
o dealing with the unexpected
o understanding process
Practicedoing m & e
(my) basic principles
o involvement of stakeholders
o definition of objectives/outcomes & associated indicators & method
o integrated into project plan from start
o properly resourced
o practical (usable) & proportionate
Monitoring & evaluation: doing m & e
(my) basic principles
o involvement of stakeholders
o definition of objectives/outcomes & associated indicators & method
o integrated into project plan from start
o properly resourced
o practical (usable) & proportionate
Monitoring & evaluation: doing m & e
an indicator“quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement.”
“ something that helps us to understand where we are, where we are going and how far we are from the goal ... They are bits of information that summarize the characteristics of systems or highlight what is happening in a system. “
Introduction to LFA: defining indicators
SMART & SPICED indicators
SMART
SpecificMeasurableAchievableRelevantTimebound
SPICED
SubjectiveParticipatory
Interpreted/communicableCross-checked
EmpoweringDiverse & disaggregated
Introduction to LFA: defining indicators
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”attributed to Albert Einsteintheoretical physicist, philosopher & author 1879-1955
Quantitative
o access & participation data (‘penetration’)o attendance & visitor data (‘reach’)o ‘audience’ characteristics (e.g. demographics)o consumer characteristicso scientific output assessment (e.g. publications)o ‘alt metrics’o amount raised/ follow-on fundingo structured opinion & feedback
Qualitative
o interviewso open questions (in questionnaire) o focus groupso ethnography o participatory researcho observationo comment/bulletin boardso visitor books
Monitoring & evaluation: rationale & overview
Theory & practice
M & E: challenges
Keeping it real: practical & proportionality
my talkMonitoring & evaluation: an introduction for practitioners
the elusive ‘impact’time frame involved
serendipity
attribution & contribution
‘ripple effects’
counter-factual
value of ‘negative’ findings
and closer to home …
getting agreement on goals, objectives, indicators & method
and resourcing m & e properly …
adequately, systematically, proportionately…
Monitoring & evaluation: rationale & overview
Theory & practice
M & E: challenges
Keeping it real: practical & proportionality
my talkMonitoring & evaluation: an introduction for practitioners
Understand stakeholder & audience requirements Be prospective: build m & e in from the start Choose right method & tailor Resourcing: ensure access to key data & information &
someone to manage this Consider option for trends & benchmarks Keep it real: be proportionate & practical – measures
can evolve Be flexible – learning is iterative & should be part of
the process Beware over-monitoring & evaluation!
SummaryMonitoring & evaluation: keeping it real
Wellcome Trust’s Indicators of ProgressOutcomes Key indicators of progress
Discoveries
Applications
Engagement
Research leaders
Research environment
Influence
1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health
3. contribute to the development of enabling technologies, products and devices
4. uptake of research into policy and practice
5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach
7. develop a cadre of research leaders8. evidence of significant career progression among those we
support
9. key contributions to the creation, development and maintenance of major research resources
10. contributions to the growth of centres of excellence
11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes
Wellcome Trust’s Indicators of ProgressOutcomes Key indicators of progress
Discoveries
Applications
Engagement
Research leaders
Research environment
Influence
1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health
3. contribute to the development of enabling technologies, products and devices
4. uptake of research into policy and practice
5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach
7. develop a cadre of research leaders8. evidence of significant career progression among those we
support
9. key contributions to the creation, development and maintenance of major research resources
10. contributions to the growth of centres of excellence
11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes
Wellcome Trust’s Indicators of ProgressOutcomes Key indicators of progress
Discoveries
Applications
Engagement
Research leaders
Research environment
Influence
1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health
3. contribute to the development of enabling technologies, products and devices
4. uptake of research into policy and practice
5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach
7. develop a cadre of research leaders8. evidence of significant career progression among those we
support
9. key contributions to the creation, development and maintenance of major research resources
10. contributions to the growth of centres of excellence
11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes
Suggested resources
Ahmed, S. & Palermo, A-G (2010) Community Engagement in Research: Frameworks for Education and Peer Review. American Journal of Public Health, 100, 8: 1380-1387
Matthew, B. & Ross, L. (2010) Research Methods: a practical guide for the social sciences. Pearson Education Limited
NORAD (2008) Results Management in Norwegian Development Cooperation: a practical guide. http://www.norad.no/en/tools-and-publications/publications/publication?key=109837
Stein, D. & Valters, C. (2012) Understanding ‘Theory of Change’ in international al development : a review of existing knowledgehttp://www2.lse.ac.uk/internationalDevelopment/research/JSRP/downloads/ToC_Lit_Review.pdf
Vogel, I. (2012) Review of the use of ‘Theory of Change’ in international development. For DfID UK
Web Center for Social Research methods http://socialresearchmethods.net
