EVALUATIONS THAT HAVE

CONTRIBUTED TO CHANGE IN

UGANDA

PRESENTED BY

WANYAMA VINCENT AND PETER WALYAULA

WORLD VISION UGANDA

Topic: Integration of outcome monitoring in M&E

system in Uganda

Overview of M&E in Uganda

1999 Poverty monitoring was introduced which provided the

foundation for assessing the impact of public policy in Uganda on

poverty and welfare.

Good quality periodic analysis of poverty trends, however, was not

matched with effective routine monitoring of Government policy

implementation.

In 2005/06 National Integrated Monitoring and Evaluation Strategy

(NIMES) was introduced to address the problem

Assessments of NIMES, the independent evaluation of Uganda’s

PEAP (2008), and reviews of public sector and financial

management reforms.

Findings from NIMES Assessments The routine monitoring of spending and results is not well

embedded across the public service.

Management information systems exist in few Ministries, and annual

sector reviews cover less than one third of sectors.

The utilization of data to strengthen performance and

accountability is generally weak.

The regular evaluation of public policies and programmes is also

sparse, with the majority commissioned and managed by

Development Partners, not Government.

Existing coverage of public investments by evaluation is estimated

at approximately ten per cent.

lessons are not being learned about which investments are

successful and which are not, and hence policy making is not

benefitting from evidence.

….Why outcome monitoring…?

Decision makers struggle to

ascertain the level of change

and use of evidence from

M&E systems to support

them in making day to day

choices.

The challenge is accelerated

by the lengthy time for

evaluation and weak

monitoring systems

The challenge is accelerated by

the lengthy time for evaluation

and weak monitoring systems

This has made onset and real

time decision making difficult for

government and development

agencies.

The current monitoring data

does not help you to understand

why something is happening at

outcome level.

Measurement of outcome

indicators in most long and short

term programs is conducted

after 2 - 5 years, as reviews or

evaluations.

And yet monitoring is the oldest

and best resourced part of the

M&E system.

….then what is outcome

monitoring…?and why the shift..? Outcome monitoring

refers to the continuous

(i.e., periodic)

monitoring of outcomes

that are supposed to

occur as a result of

program operation (i.e.,

what conditions are

supposed to change as a

result of the program?).

Monitoring of indicators

or factors that are listed

in the program impact

theory.

Outcome monitoring

measures what is supposed to

result from the program.

The argument presented here

is as a result of increased

demand from decision makers

(Donors, Managers,

governments etc ) to a shift

from the conversational

monitoring mechanisms to a

different kind of monitoring,

the monitoring of selected

program outcome indicators

that can be collected

repeatedly over time.

..Why shift to Outcome monitoring..?

The focus here is to measure and assess on continuing

“performance” for selected program indicators, in the

programme impact theory and communicate with partners

for decision making annually.

It also involves applying the four steps in the logic of

evaluation to the outcome monitoring data

1. select dimensions of merit,

2. set standards on those dimensions,

3. collect performance data

4. Integrate the results into evaluative judgments. This involves

understanding of Program Outcomes, Identifying Relevant

Outcomes indicators and definitions, Measuring Program

Outcomes and Monitoring Program Outcomes.

SELECTED CASES IN AFRICA

Ghana & Kenya annual reports

main product - produces 2

reports, annual progress report

against indicators (NIMES)&

public expenditure reviews

South Africa has quarterly

performance reporting by

department and against priority

outcomes & dept annual

reports

Uganda – regular monitoring six

monthly reporting directed to

politician

Uganda government policy

framework moved to regular

monitoring systems linked to

six monthly reporting directed

at politicians

Ghana 70% of the M&E

budget for 2010 was

reported as being spent on

monitoring activities alone

(NDPC 2011).

Uganda and South Africa high

demand for monitoring

evidence (increased

performance in cabinet)

Benin – strengthen evaluation

Senegal – systematic use of

existing initiatives

South Africa and Uganda

institutionalize monitoring to

feed into executive decision

making (Porter, S. &

Goldman,i.,2013 A growing

demand for M&E in Africa’,)

Case of World Vision Uganda

Outcome Monitoring began in 2012

Integrated into the M&E system

Piloted a cross 54 Area development programmes

Conducted annually

Use LQAS methodology

Results are used to inform annual planning process and research

Policy advocacy

Management uses the information for decision making

Staff capacity building

Integrated with new technologies like mobile devices, GIS etc

Outcome monitoring results is used in the Evaluation and reviews

e.g. All evaluations conducted in the last 3 years

CASE STUDY

KIRYANGA EVALUATION

Children (boys and girls) completing primary School

20%) and 14.2% of the boys and girls who enroll in primary school

complete. This means only 2 out of 10 boys who enroll in primary

school complete and less 2 out of 10 of the girls complete

The completion rate for both girls and boys is only 17.2%, that is to

say, less 2 out 10 boys and girls who enroll in primary school

complete.

80% and 85.5% of the boys and girls respectively who enroll in

primary school dropout.

9 out of 10 of the girls who enroll in primary school dropout

8 out 10 of the boy’s dropout.

The dropout rate for both boys and girls was 82.8%.

A trend Analysis of Children enrolled in

primary schools

2008-2014

1500

1600

1700

1800

1900

2000

2100

2200

2008 2009 2010 2011 2012 2013 2014

Total number of boys and girls

enrolled

Year

Male

Female

2008 to 2014

shows that

enrollment of

boys and girls

keeps rising and

falling.

Between 2009

and 2014,

enrolment

increased by 5.9%

while between

2013 and 2014, it

declined by 6.5%.

Enrollment and Completion at Kabamba Primary School

Reasons given for Low

completion and Retention

Rates

Early marriage

Child labor

Teenage pregnancy

Negative attitude of

parents/guardians towards the

education of their children

Some parents do not provide

scholastic materials and other

school requirements for their

children

Constant migration of some

parents

0

20

40

60

80

100

120

140

160

180

2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Pu

pils

pe

r C

lass

Year

P.1

P.2

P.3

P.4

P.5

P.6

P.7

Education indicators

Functional Literacy

57.2% of the pupils were

functionally literate

(14.6% could read a

story; another 14.6%

story plus

comprehension;

28% local material plus

comprehension). More

boys, 36.6% than girls,

20.8% were functionally

literate.

Proportion of children

who are functionally

literate by age 13 years

was only 20%.

Early Grade Reading

Assessment

Early grade reading assessment

conducted among primary two

and three pupils show that:

72.9% could not read a single

word in the first line of the

simple story used in the

assessment;

11.6% could read less than 45

words per minute; and only

15.6% could 45 or more

words per minute.

Education indicators

Children Promoted from one class

to another scoring an average of

50% and above

Promotional exam results show that

only 27.3% obtained an average mark

of 50% and above from all subjects

A larger proportion of children, 72.7%

scored below the 50% average mark.

Disaggregation by gender shows girls

performing slightly better the boys,

15.1% the girls obtained an average

mark of 50% and above, against 12.6%

of the boys; and more ( 40.3%) boys

than girls ( 31.9%) scored below 50%

average mark.

Pupils passing in Division

one and two in Primary

Leaving Examinations

Proportion of pupils passing in

division one and two declined

by 2.2% from 28.7% in 2009 to

26.5% in 2013.

This performance is far below

the national performance of

53.5% of pupils who obtained

division one and two in the

PLE of 2013.

These findings suggest that the

programme interventions had

no impact on pupils’

performance in PLE.

Outcome monitoring data

% of the boys and girls

passing in grades 1 &

11. 2012

Division/grade 1

0.3% and 2013 was

3%

Division/grade II

2012 25.3% and 33%

in 2013

Proportion of children

who are functionally

literate by age 13

years 25% in 2012

and 26% in 2013

METHODOLOGY

The study used a mix of both

quantitative and qualitative

survey techniques. This study

adopted the use of a cheap and

cost effective methodology

called Lot Quality Assurance

Sampling (LQAS). The method

involved taking a small random

sample of 19 from each set of

items from a population.

For example, LQAS designs have

been shown to provide a

statistically rigorous alternative

to the more time-consuming and

expensive SMART survey

method (also a two-stage

cluster-sampled design) that is

commonly used.

The LQAS survey method

can provide an alternative to

traditional cluster sampled

surveys.

Data was analyzed using a

systematic process of

tabulation of response in the

tabular sheet and later

weighted.GIS was also used

to locate points of

interventions.

Descriptive statistics were

presented in form of

frequencies per indicator. For

qualitative data, review of

annual reports, M&E plans,

Design documents and FGDs

were conducted.

Benefits of Outcome monitoring when integrated

into M&E system

Ensures close focus of

outcome targets in day-to-day

implementation

Link continuous findings on

monitoring data to outcome

and evaluation

Strengthen evidence based

targeting and planning

Reallocation of resources

Replication of good practices

Policy influence

Timing

Evidence based programming

Thanks

THANK YOU

&

BE BLESSED !!!