Monitoring, Evaluation,

Accountability &

Learning

Key objectives of this session:■ Understand key components of the

global MEAL framework■ Share feedback on the global Voice

MEAL framework■ Work on a first draft of the global

Voice MEAL plan■ Address questions and concerns

Let’s review some concepts

Output Input Outcome

Monitoring Evaluation

The financial, human, and materialResources used for each intervention

Measurable results that help quantify the reach and scale of the Voice programme

Short, medium and longer term changes that Voice plans to achieve in the areas of capacity strengthening and linking and learning

IndicatorsA qualitative or quantitative factor or variable that provides a simple and reliable means to measure change

The continuing function that uses systematic collection of data to provide progress against objectives

The systematic and objective assessment of an on-going or completed project, program or policy, its design, implementation and results.

Program vs. Project

• A set of strategically aligned, mutually reinforcing interventions – that contributes to sustained, positive change or several target group members’ lives.

• A program = several projects which can begin and end within the lifetime of a program.

well defined target group

period for implementation

one / several grantees or

partners

Funding is allocated at this level

Contractual relationship

PROGRAM PROJECT OR GRANT

Understanding Theories of Change

How does change happen?

“Theories of change are the ideas and

hypotheses (‘theories’) people and organisations have about how change

happens. These theories can be conscious or unconscious and are based on personal

beliefs, assumptions and a necessarily limited, personal perception of reality.

ToC is not a diagram!

■ On-going reflection• Wider analysis of how

change comes about.• Challenging• Focused on what we

want• Complex

■ Academic research■ Linear■ Static■ Focused on what we plan■ Not (necessarirly) reality

What can we use it for?

■A way to clearly communicate (often complex) pathways of change we envisage

■Allows us to acknowledge and question our assumptions of how change does and should occur

■It further clarifies how we think about power and its role in the specific change we are working towards

■Provides a basis for evaluation, learning and accountability

Common mistakes we make

1. We overcomplicate it 2. We develop a theory and never go back to it3. We are restricted by the linear programme log

frame 4. We fail to acknowledge our implicit existing

theory & ideology5. We don’t look beyond CSOs when mapping

(binary world of civil society and state)6. We plan for steady state but we live responding

to opportunities7. We assume the system is static until we arrive8. We don’t keep a record of what’s happened

How does change happen?

Thinking about How Change could happen in relation to a particular issue

What needs to change? At what level?What actors and factors in the external context help or hinder change?

Who has the power to influence change either positively or negatively

What are critical events and opportunities in the way?This thinking goes beyond your own intervention and considers all aspects of addressing identified issues

Theory of Change Cycle

A Voice global Theory of Change (1)

What needs to be in place before deciding on the pathway:• Baseline studies x10• Current state of affairs• SMART objectives• power and gender analysis x10• power mapping (or stakeholder analysis)• Organizational assessments

A Voice global Theory of Change (2)

MEAL framework: Background

■ Basis: Tender document

■ Linked to the Theory of Change and main activities

■ MEAL outcome areas fall under pre-selected IATI outcome areas

■ Qualitative & Quantitative indicators used

Overall outcome goal

VOICE will strengthen the capacity for lobby and advocacy of civil society organisations and representatives of the most marginalised and discriminated groups in order to empower these groups and to have their voices heard and respected, using innovative ideas in the field of capacity strengthening and in the field of lobby and advocacy.

From Theory of Change to Outcome areas

Outcome areas Outcome indicators (quantitative)

1. (Informal) groups have capacity to work on their own empowerment through building safe spaces, creating confidence and skills

400 groups have presented their collective demands to external stakeholders

50% of targeted groups indicate to be able to have safe spaces to convene and work

80% groups indicate an increased confidence to raise their voice

2. (Informal) groups, organisations and networks have capacity to build alliances and networks with a variety of stakeholders, to amplify the voice of marginalised and discriminated groups.

76 alliances and coalitions indicating to initiate activities based on joint agenda

152 publications (media outlets, relevant policy documents etc) supportive to the position of marginalised groups and their change agenda.

3. (Informal) groups, organisations and networks have capacity to use a range of tools and instruments to influence key decision makers

groups, organisations and networks report a benefit (being able to influence) by using a diverse set of advocacy tools (400 groups, 128 organisations and 76 networks)

Targeted outcome area 1

Targeted outcome 1.1: (Informal) groups have capacity to work on their own empowerment through building safe spaces, creating confidence and skills

Output Output indicator

1.1.1 Programme staff and partner organizations have facilitated the development of the agenda of change by marginalised and discriminated groups.

400 agendas for change developed

1.1.2 Marginalised and discriminated groups, with support from partner organizations, have implemented their agenda for change

400 development agendas funded

1.1.3 Marginalised and discriminated groups incorporate new approaches in realising their agenda for change

200 groups indicating use of new approaches

Targeted outcome area 2

Targeted outcome 1.2: (Informal) groups, organisations and networks have capacity to build alliances and networks with a variety of stakeholders, to amplify the voice of marginalised and discriminated groups.

Output Output indicator1.2.1 (Informal) groups, organisations,

networks have defined a joint agenda and campaigns strategy

76 joint campaigns defined

1.2.2 (Informal) groups, organisations and networks have implemented a campaign to raise public support for the position and demands of marginalised and discriminated groups.

64 public campaigns funded by VOICE

Targeted outcome area 3

Targeted outcome 1.3: (Informal) groups, organisations and networks have capacity to use a range of tools and instruments to influence key decision makers

Output Output indicator

1.3.1 (Informal) groups, organisations and networks have defined a plan to strengthen their capacity to influence decision makers.

114 capacity strengthening plans defined

1.3.2 VOICE has allocated grants to support the implementation of influencing strategies by marginalised and discriminated groups

# influencing and sudden opportunity grants provided by VOICE (64 influencing grants and 50 sudden opportunity grants)

1.3.3 VOICE has provided access to expert support and to platforms of social innovators and influencing tools

20 experts and 6 platforms facilitated by Voice

Gender, women’s rights & indicators

■ Gender un-aware/blind

■ Gender neutral

■ Gender sensitive

■ Gender transformative

Gender unaware, neutral, sensitive or transformative?

■ 50% of primary change agents express satisfaction with government accountability mechanisms Priorities of people disabilities are included in community projects funded by the central government

■ 80% of indigenous people have access to safe provisional housing and food.■ 80% of Boys and Girls between ages of 10 and 15 have access to HIV prevention information.■ 200 Small scale farmers men and women are selling their produce in local markets ■ % of women participating in project activities■ 300 women-lead households increase their income by 40% through improved market linkages.■ % of female students living in Spring City who report an increased sense of personal

security/safety in public spaces. ■ 30% increase in women running for public office in local electoral processes. ■ At least 200 women report that home workload is equally shared with men and have more time

for themselves.■ 80% of women credit unions report being part of decision-making spaces and demonstrate

effective participation.■ Public opinion polls indicate that at least 50% of men strongly agree with women´s rights and

capacities to run for public office in local and national elections.■ At least 30% of local budgeted earmarked to support women rights and prevention of VAW

initiatives.

■

Links between Program and Project, Local and Global

■

Voice overall outcome goal

Grant

Grant

Grant

Grant

Grant

Grant

Voice focus

countryVoice focus

country

Hivos and Oxfam change goals/ objectives

Country change

goals/ objectivesCountry change

goals/ objectives

Group work1. How can we assess the quality of our targeted

outcomes? What are your suggestions for qualitative indicators (max 5) per targeted outcome?

2. What are the key assumptions and risks associated with each one of the outcome areas?

In teams of 3 – 1 team per outcome area

1. MEAL planningWhat requirements and within which timeframe?

MEAL matrix: Key elements

■ Establishing a baseline■ Define time-bound activities linked to

outputs outcomes (planned/unplanned)

■ Data collection plan■ Monitoring activities■ Learning and project reviews■ Regular checks on ToC assumptions■ Feedback loops

Data collection & analysis

1. Disaggregate variables2. Identify unit of analysis3. Select / prioritize methods4. Establish frequency5. Define responsibilities6. Plan for analysis 7. General revision

Data management: Key areas of consideration

Source: What are we collecting?• Collection: Who collects this data, from where, and how often?

Collation and Storage• How are data aggregated? And by whom?• Where is the data stored?• Who of the project team is responsible for managing all the data?

Analysis• Possible opportunities to transform the data into more meaningful information and thus for further

review. • Who would be involved in this review and transformation? • Other pieces of information available, besides your own collected data to help you with verification

Reporting• Who will do the write up of the analysis?• To whom will this information be reported?• Use• How can this information be used to make informed decisions?

Gathering responsible data responsibly!

1. Voluntary participation and informed consent

2. Do no harm:3. Confidentiality & Anonymity4. Health and Safety5. Purposeful use of data 6. Good use of funds7. Children-sensitive data?

MEAL framework in practice

Inception:Baseline

Call for proposalsBudget for MEAL

Partners 'assessment

Planning and Implementation:

Identify long term goal and outcomes

Define lines of actionIdentify assumptions

Define reach and scaleRisk analysis

BudgetPlan for exit

Review indicatorsHarvest outcomes

Closure:Exit/handover or scale

up?Capturing and sharing

learning for future programme work

Oxfam’s Common Approach (CAMSA)

Monitoring (1)Projects with a life span that is less than a year

■ Capture reflections on the learning journey

■ No prescriptive format – use your best judgment

■ Recommended areas to monitor: challenges/successes, level and type of engagement, socialization of knowledge

Place your screenshot here

Monitoring (2)Bi-annual light touch reporting (project min. 12 months)

■ Highlights on the evolution of the context, (emerging) lessons on the assumptions underlying ToC

■ Progress on outputs – data should be disaggregated by sex

■ 3 key lessons learnt (successes and challenges) to be used as input to the learning process.

Place your screenshot here

Monitoring (3)Annual reporting – projects min. 12 months

■ Evolution of the context, (emerging) lessons on the assumptions underlying ToC,■ Update of the risk analysis ■ Progress on outcome areas and outputs – data should be disaggregated by gender■ Key findings/learning from monitoring, learning or impact measurement methodology

implemented (outcome harvesting, stories of change).

Reporting levels

■ Milestone grant process reporting

■ Annual grant reporting (grants >= 1 year)

■ Light-touch learning report (grants > 1 year)

■ 6 – monthly internal reporting: # grants (type, amount, themes), activities; main achievements/challenges

■ Time-bound financial reporting: salaries; use of budget lines

■ Annual reporting on outcome indicators

Reporting formats for smaller grants: potential questions

Please summarise the activities conducted in the reporting period■ What key successes have you experienced so far? ■ What challenges has this project faced?■ What challenges are foreseen and how are you mitigating them?

Target Groups■ Explain how likely the intervention is to change the situation of the selected

target group(s)

Organisation, Evaluation and Learning■ Please share any updates to or concerns about the expected results and

timeframes described in the proposal. Particularly if there have been any major changes to outputs or activities.

■ What systems were put into place to monitor the implementation and learn from its results?

■ What support can Voice MEAL teams provide you? (Optional)■ What updates would you like to communicate regarding the key

achievements/messages of the project and how would you do it? (Optional)

Reporting timeline

January:Call for

proposalsDonor report

for ongoing/closin

g grants

July: Bi-annual (light touch) internal

reporting

September: Planning

Indicators reviewNew call

December: December:

Internal yearly reporting on

- Status- Outcome indicators- Learning

Hivos internal

reporting: M

arch/

April

Oxfam output

reporting (direct +

indirect #): March

Who are we accountable to?

Type Minimum standard

TransparencyEssential information on VOICE will be publicly made available and is accessible

through the website, reports and publications. Proactive information sessions will be held specifically for groups that are hard to reach.

Participation

The programme will enable (with funding and adaptive tools) that women and men, beneficiaries of the programme, as well as their organizations define their agendas for

change that will serve as the basis for the allocation of grants.

Local actors will be part of the  communities of stakeholders, that participate in setting priorities for VOICE in each of the participating countries. 

Monitoring, Evaluation, Learning

Groups and organizations will define indicators for success and learning questions and are actively engaged in measuring progress and sharing answers and insights.

Feedback

The program will have a feedback and complaints mechanism in place and will proactively encourage stakeholders to give feedback. This mechanisms will be

explained on the website. All feedback will be recorded and responded to.

Mechanisms deal with positive and negative feedback in addition to complaints including response

Some more group work!1. How do we make our MEAL framework more inclusive, ensuring that we do raise the voice of the marginalized and discriminated groups?

2. What methods can we use to feed the results back to our grantees (especially informal organizations) and target groups?

Key questions in the back of our head:■ What information do we need?

■ At which level do we need to look for it? ■ What method will we use to collect it?

■ Who will be responsible for generating the information? ■ How frequently will we collect the information?

(3 groups x 3 participants per group)

Unplanned outcomes

Planned outcomes

Longer term change

Measuring longer term (transformative) changes

Res

ults

Fr

amew

ork

Sto

ries

of

Cha

nge;

O

utco

me

Har

vest

ing

Sum

mat

ive

eval

uatio

n

Good practice! Consider developing criteria to identify stakeholders that should participate in

different stages of the evaluation. The criteria should prompt consideration of how to ensure that

participation is representative of the stakeholder group and equity e.g. for participation of women

Evaluation management: Roles and Responsibilities in Voice

What to plan for? Who will be responsible?

Evaluation plan and budget Programme manager

Scoping and mandate of the evaluation Programme manager

Planning of the evaluation Programme manager

Implementation of the evaluation (process – baseline, midterm,

endterm)

MEAL Officer global for final evaluation and contracting authority for the mid-term review

Quality assessment and management response Programme manager

Using the outcomes of the evaluation Programme manager

Are we talking about Impact?

Impact vs. continuous, transformative change

https://www.youtube.com/watch?v=5f4rNEsyEYY

Thanks!Any questions?

Catch me on skype @ lina.kagkliOr email me at: lina.kagkli@voice.global