Developing a common Monitoring and Evaluation (M&E) framework for the IASC Guidelines on Mental...

Developing a common Monitoring and Evaluation (M&E) framework for the IASC

Guidelines on Mental Health and Psychosocial Support (MHPSS) in

emergency settings

Jura Augustinavicius, MSc MHS

Doctoral Student

Johns Hopkins Bloomberg School of Public Health

Outline

• Key Monitoring and Evaluation (M&E) concepts• Main components of Logical Frameworks

(Logframes)• IASC• Need for a common M&E framework• Review

– Aims– Introduction of Draft I of the Common M&E Framework– Methods– Results

• Conclusions

Key Concepts: M&E

Functions of M&E:

1. Process Evaluations: How well was the project implemented?

2. Monitoring outputs and outcomes: Did changes occur in the target population?

3. Impact evaluation: Can observed changes be attributed to the project or programme (i.e. cause and effect)?

Key Concepts: Logframes

• A systematic tool for designing, planning, implementing, and monitoring and evaluating a project or programme– Can help organize thinking – Identify and assess risks inherent in project

design and implementation– Measure project progress– Develop consensus and communicate a

project’s intent and strategy

PROJECT STRATEGY

GOAL: The overall good to which the programme contributes (and to which other factors and programmes may also contribute)

PROJECT STRATEGY


OUTCOMES: The specific parts of the overall goal that the project will achieve and that the project is held responsible for

PROJECT STRATEGY


OUTCOME: The specific part of the overall goal that the project will achieve and that the project is held responsible for

OUTPUTS: The project intervention; Direct and immediate products of activities

PROJECT STRATEGY




ACTIVITIES: What people who are employed by the project do to assist the community

PROJECT STRATEGY





INPUTS: Major resources required to carry out activities

PROJECT STRATEGY

ASSUMPTIONS: Variables that are outside of the control of the project but that affect the success of the project

• Use “if, then” logic

• Should be generated based on past experience with similar programs






PROJECT STRATEGY

INDICATORS: Used to measure progress towards achieving the project strategy







PROJECT STRATEGY

INDICATORS: Used to measure progress towards achieving the project strategy

MEANS OF VERIFICATION: Sources of indicator data







IASC

• IASC guidelines were developed in 2007

• Published consensus on MHPSS in emergencies

• IASC MHPSS Reference Group continues to meet to disseminate and implement the guidelines and to develop MHPSS products

• 30+ member organization

A Common M&E Framework

• Systematic M&E can offer a means to understand the relative benefits of MHPSS interventions

• Consolidate goals, outcomes, and indicators for MHPSS activities across organizations

• Offers a common platform from which to advocate for prioritizing MHPSS in humanitarian settings

Aim

To identify the most commonly applied strategies to monitor and evaluate MHPSS interventions in LMICs using a logframe approach

Methods1. Review of Logframes and Theories of Change

Documents collected from IASC RG MHPSS member organizations across humanitarian sectors and anonymized



2. Narrative Review

EMBASE, PILOTS, PsycInfo, PubMed/MEDLINE, and WHO regional databases



2. Narrative Review

EMBASE, PILOTS, PsycInfo, PubMed/MEDLINE, and WHO regional databases

• Standardized instructions for double data extraction of: • Goals • Outcomes• Indicators• Means of Verification

• Top-down: Mapping outcomes and indicators to outcomes from the preliminary IASC RG framework

• Bottom-up: Qualitative (thematic) analysis to identify common themes at each logframe level

Goal: Enhanced mental health and psychosocial wellbeing/reduced psychological and social suffering

Outcomes:

Community-focused (1) Emergency responses are safe, dignified, participatory, community owned, socially and culturally acceptable

(2) People are safe, protected and human rights violations are addressed

(3) Existing social, community, and family systems promote an environment for recovery and human development

Person-focused (4) Communities and families support people with mental health and psychosocial problems

(5) People with mental health and psychosocial problems access appropriate focused care

Underlying core principles: 1. Human rights and equity; 2. Participation; 3. Do no harm; 4. Integrated services and supports; 5. Building on available resources and capacities; 6. Multi-layered supports

Developed based on input from workshops in Sri Lanka, Lebanon, and Uganda

Results: Logframe & TOC Review

39 Documents reviewed• Logframes

(84.6%)

Results: Logframe & TOC Review

Results: Narrative Review

88 Articles and Book Chapters• Studies published between

1992-2013

Study Population• Children and adolescents

(31%)• Adults (32%)• Combination of children

and adults (23%)• Unknown (14%)

Results: Narrative Review

Qualitative Analysis Results

• Theme 1: Increasing resilience, promoting social wellbeing, and preventing mental health and psychosocial problems

• Theme 2: Reducing ill mental health, psychosocial symptoms, and associated functional impairment through care

• Theme 3: Building capacity to identify, intervene on, and monitor MHPSS problems

• Theme 4: Promoting optimal human development within existing social systems

• Theme 5: Macro level goals that seek to build peace between groups after conflict and to address structural problems within societies

• Theme 6: Protection of vulnerable groups, such as women, children, the elderly, and people with disabilities

Conclusions

• The current draft of the common MHPSS M&E framework can be applied to existing documents and peer-reviewed literature

• Confusion with regard to defining goal and outcomes

• Although psychometrically sounds means of verification are published in the peer-reviewed literature, these are not commonly applied to logframes or theories of change

IASC MHPSS Reference Group

Wietse Tol

Ananda Galappatti

Saji Thomas

Alison Schafer

Margriet Blaauw

Mark van Ommeren

Johns Hopkins Bloomberg School of Public Health

M. Claire Greene

Annisa Harsha

Megan Cherewick

Jillian Emerson

Katharine Shelley

Amanda Samson

Srihari Cattamanchi

Saumya Singh

Acknowledgements

With funding from UNICEF

Additional Slides

Key Concepts: M&E

Monitoring: A continuing function that aims primarily to provide the management and main stakeholders of an ongoing intervention with early indicators of progress, or lack thereof, in the achievement of results

Key Concepts: M&E

Evaluation: A selective exercise that attempts to systematically and objectively assess progress towards and the achievement of a health outcome

Key Concepts: M&EProgramme Monitoring Programme Evaluation

What is it?• Assessment of whether a

project is being implemented as planned

Assessment of whether:• The project achieved the

objectives/intended results• The project provided value

and made a difference

Why do it?

When is it done?

What is assessed?







Why do it?• To determine what changes

are needing during the project• To decide what activities to

repeat in future projects

When is it done?

What is assessed?










When is it done?

• During the life of the project• Frequently or continuously

• At the beginning and end of a project (occasionally at the midpoint)

What is assessed?










When is it done?

• During the life of the project• Frequently or continuously

• At the beginning and end of a project (occasionally at the midpoint)

What is assessed?

• Resource amounts and their distribution

• Delivery, distribution, and quantity of services

• Costs and expenditures

• Actual changes relative to intended changes

• Value of project activities and achievements

Developing a common Monitoring and Evaluation (M&E) framework for the IASC Guidelines on Mental...

Documents

Transcript of Developing a common Monitoring and Evaluation (M&E) framework for the IASC Guidelines on Mental...