040-6363cover 10/10/03 10:04 AM Page 1

T H E W O R L D B A N K O P E R A T I O N S E V A L U A T I O N D E P A R T M E N T

Growing

concerns about poor value for money from Uganda’s public services are focusing attention on the need for better understanding of development effectiveness what works, what does not, in which contexts, and why. This issue is being addressed by efforts to strengthen Uganda’s monitoring and evaluation (M&E) systems.

The Development of Monitoring and Evaluation Capacities to Improve Government Performance in Uganda

ECD Working Paper Series ♦ 10 Arild O. Hauge

www.worldbank.org/oed

October 2003 The World Bank

Copyright 2003 Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) Email: eline@worldbank.org Telephone: 202-473-4497 Facsimile: 202-522-3125 Evaluation Capacity Development (ECD) helps build sound governance in countries—improving transparency and building a performance culture within governments to support better management and policymaking, including the budget process—through support for the creation or strengthening of national/sectoral monitoring and evaluation systems. OED aims to identify and develop good-practice approaches in countries, and to share the growing body of experience with ECD efforts. The OED Working Paper series disseminates the findings of work in progress to encourage the exchange of ideas about enhancing development effectiveness through evaluation. An objective of the series is to get the findings out quickly, even if the presentations are somewhat informal. The findings, interpretations, opinions, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the Operations Evaluation Department or any other unit of the World Bank, its Executive Directors, or the countries they represent.

CONTENTS

Foreword iv Abbreviations v Executive Summary vi 1. Introduction 1 2. Main Features of Uganda’s M&E Systems 2 3. What Contribution Is M&E Making to National Development? 6 4. What Challenges Remain? 9 5. Lessons Learned from Uganda’s M&E Experience 13 6. Conclusions 16 Annex A: Poverty Monitoring and Evaluation Strategy: Priority

Poverty and Sector Indicators 17 Annex B: Summary of OED M&E Diagnosis Findings, 2001 18 Annex C: OED’s Contribution to M&E Capacity Development (ECD)

in Uganda 19 Annex D: M&E in Uganda PRSO3 Draft Policy Matrix

(as of April 2003) 25 Annex E: Select Findings from the Study of M&E in Uganda’s Health, Education, and Water Sectors 28 References 31

FOREWORD As part of its mandate, the World Bank’s Operations Evaluation Department (OED) has responsibility for encouraging and assisting developing member countries to build effective monitoring and evaluation (M&E) capacities and systems. OED plays a catalytic role, including support to a small number of countries with the intention that they will develop good-practice M&E approaches that can be adapted by other countries to suit their particular circumstances. In this context OED has provided active support for Uganda since 2000. The Ugandan government is committed to improving the development effectiveness of its activities by means such as the achievement of a stronger performance orientation in public management. Its efforts in this area, and those of many of its development partners, have been framed in the context of the Poverty Reduction Strategy. Uganda has pioneered access to the debt write-off associated with the Highly Indebted Poor Country initiative, and it has also pioneered the Poverty Reduction Support Operation (formerly the Poverty Reduction Support Credit), a type of programmatic lending. This report provides a follow-up to a diagnosis of Uganda’s M&E systems and functions, which was conducted in 2000 and published in January 2001. This latest report attempts to identify the contribution that M&E is making to national development and examine the remaining challenges to institutionalising M&E in a sustainable manner. A number of lessons from Uganda’s efforts in this area are also discussed. A draft version of the paper was presented to OED and Bank staff in Washington, D.C., on June 1–2, 2003, and to members of the Uganda Evaluation Association in Kampala on June 19, 2003. We are grateful for comments received from a number of people, including Ismail Barugahara, Poul Engeborg-Pederson, Satu Kahkonen, Margaret Kakande, Sukai Prom-Jackson, and Peter Ssentongo. The strong support of Satu Kahkonen, of the Bank’s Africa Region, has been most valuable. In addition, the generous financial support provided by the Ministry of Foreign Affairs of the Norwegian Government is gratefully acknowledged. The report was prepared by an OED consultant, Arild O. Hauge. The task manager was Keith Mackay (OED).

Osvaldo Feinstein Manager

Partnerships and Knowledge Programs, OEDPK

iv

ABBREVIATIONS

AfrEA African Evaluation Association AIDS Acquired Immunodeficiency

syndrome BFP Budget framework paper CAPEP Capacity and Performance

Enhancement Project CDF Comprehensive development

framework CSO Civil society organization DEI Directorate of Ethics and Integrity DFID Department for International

Development (United Kingdom) DWD Directorate of Water Development

(of the Ministry of Water, Lands and Environment

ECD (Monitoring and) evaluation capacity development

GDP Gross domestic product GoU Government of Uganda HIPC Heavily indebted poor country HIV Human immunodeficiency virus HPPG Harmonized participatory planning

guide IFMIS Integrated financial management

system IPDET International Program for

Development Evaluation Training JLO Justice, Law and Order LGDP Local Government Development

Programme LGFC Local Government Finance

Commission LGs Local governments M&E Monitoring and evaluation MDGs Millennium Development Goals MIS Management information system MOES Ministry of Education and Sports MOFPED Ministry of Finance, Planning and

Economic Development

MOH Ministry of Health MOLG Ministry of Local Government MOPS Ministry of Public Service MOWLE Ministry of Water, Lands and

Environment MTEF Medium term expenditure

framework NGO Non-governmental

organization NSDS National service delivery

survey ODA Official development

assistance OED Operations Evaluation

Department OPM Office of the Prime Minister PAF Poverty Action Fund PEAP Poverty Eradication Action

Plan PETS Public expenditure tracking

survey PMES Poverty Monitoring and

Evaluation Strategy PRSC Poverty Reduction Support

Credit PRSO Poverty Reduction Support

Operation PRSP Poverty Reduction Strategy

Paper ROM Results oriented management SWAP Sector-wide approach SWG Sector working group TA technical assistance UBOS Uganda Bureau of Statistics UEA Uganda Evaluation AssociationUPPAP Uganda Participatory Poverty

Assessment Process USh Uganda shilling

v

EXECUTIVE SUMMARY There is a growing awareness that Uganda’s progress with poverty reduction does not match the rate of increase in budget resources for the social sectors; there are indications of poor effectiveness and value-for-money in public service delivery. These concerns are focusing attention on the priority for a better understanding of development effectivenesswhat works, what does not, in which contexts, and why. Uganda’s monitoring and evaluation (M&E) systems have the potential to provide this much-needed understanding. A large volume of information is currently produced by different M&E systems and practices that have been established in Ugandaemanating from domestic and external concerns with accountability, governance, public sector reform, and financial management. But the disparate information flows can create confusion about goals and complicate policy analysis. Parallel systems lead to duplication and waste in data collection, and the M&E workload diverts attention away from productive service delivery. The M&E systems also sometimes reward managers for good paperwork rather than contribution to poverty eradication. Although arguably documenting compliance with nominal accountability rules, Uganda’s M&E systems need to increase the emphasis on the results that follow from public action. M&E data are often of poor quality, with missing, inaccurate, or outdated information. The distinction between observed reality and what is hoped-for is blurred. In this environment, donorswhen each brings a different set of rules and requirementshave been part of the M&E problem rather than part of its solution. The Uganda experience helps to dispel the notion that increased M&E, in and of its own, will lead to improved results orientation. What Uganda needs is not more, but better, M&E. This priority is now being addressed by the Poverty Reduction Support Credit (PRSC)recently renamed the Poverty Reduction Support Operation (PRSO)process, which has become the vehicle for a systems approach to M&E development in Uganda. By marshalling donors toward budget support, it is inducing a greater reliance on the government’s own M&E systems and practices. Equally important, the PRSCs/PRSOs have explicitly identified a series of actions to address four major M&E issues that remain. First, whereas M&E systems produce fairly accurate measures of expenditures on the one hand and poverty on the other, there are fewer systematic data on the intervening levels: that is, outputs, the targeting effectiveness of government service delivery, and outcomes. Second, M&E is not geared toward understanding causality and attribution between the stages of development change. Third, incentives are tied to compliance with reporting requirements, rather than the underlying performance revealed by M&E. Fourth, the requirements that accompany M&E arrangements for different national and donor funding schemes result in a complex and formidable burden of guideline familiarization, inspection activity, and data collection and reporting. It can be argued that the growing attention to M&E is already helping to bring greater rationality to public finances and development managementby providing a more evidence-based foundation for policy, budgeting, and operational management. M&E-related initiatives are helping instill an orientation toward the government’s poverty reduction goals (as

vi

expressed in its Poverty Eradication Action Planits poverty reduction strategy. They are also helping to reduce the government’s transaction costs of donor coordination; and they are increasingly allowing civil society and Parliament to assume a more meaningful role in public affairs. A key lesson from Uganda is that technical constraints, such as skills, manuals, and logistical resources, are not the predominant bottleneck to capacity. M&E will flourish where there is strong demand from policymakers and advisers for M&E information, where managers seek an empirical foundation for decision-making, where the practice of M&E follows as a consequence of incentives embedded in public service systems, where rewards and sanctions are guided by achievement of results, and where managers collectively perceive a self-interest in adopting tools of continuous assessment and learning. Making further headway with M&E will necessarily require coordination and operational action on several fronts. With an M&E working group established under the PRSC/PRSO, coordination is now being addressed, paving the way for the remaining M&E challenges to be addressed systematically.

vii

1. INTRODUCTION Donor and developing country public management policymakers, analysts and practitioners alike, are closely watching Uganda. Over the last decade, the country has achieved a rate of growth unparalleled in the Sub-Saharan region. Uganda has pioneered access to the Heavily Indebted Poor Country (HIPC) initiative debt write-off and the Poverty Reduction Support Operation (PRSO) modality(PRSOs involve grants, and they recently replaced Poverty Reduction Support Credits (PRSCs), the earlier form of programmatic support for Uganda’s poverty reduction strategy. Substantial and multifaceted reforms in public sector management have been undertaken. Parallel to the unfolding reform in Uganda, increased international attention is being given to the role that monitoring and evaluation (M&E) plays within public management results orientation in general and development effectiveness in particular. Moreover, in Uganda M&E has been identified as a priority area of cross-cutting public sector reform within the policy matrix of the PRSO series of operations that have been planned and implemented since early 2000. This paper reflects lessons learned from work undertaken as part of Operations and Evaluation Department (OED) support for national M&E capacity development within the PRSO context. The finalization of the paper has greatly benefited from discussions held with, and comments provided by, World Bank staff and Ugandan stakeholders. The objective of the paper is to take stock of the M&E initiatives and capacity development efforts that have taken place in Uganda, with a particular reference to those lessons that emerge as relevant to the wider development community. The analysis takes the functioning of M&E within the Uganda public sector, rather than the needs and requirements of the external donor community, as its main point of departure. The objective of M&E is seen as the improvement of the performance and effectiveness of government and its public service delivery system. M&E relates to more than the technical aspects of data collection: it ultimately derives its value not from the availability of accurate data per se, but from the way in which they are used to improve decision-making. M&E functions when it supplies managers with a flow of reliable information and analysis about what works and what doesn’tas a basis for policy recommendations, optimising resource allocation, refining institutional strategies, changing the mix of public service products or suppliers, assessing performance of staff, or underpinning accountability requirements. Because M&E addresses the performance of governments in responding to their citizens’ needs and priorities, it is ultimately an issue of good public governance.

1

2. MAIN FEATURES OF UGANDA’S M&E SYSTEMS If the output of M&E capacity development is more comprehensive production and availability of M&E information and findings, Uganda has made substantial progress. A number of new M&E systems and practices that enable production of data and analysis have been established within the last decade. These have often arisen from concerns associated with, for example, accountability (value for money, malpractice, and audit follow-up), governance (participation, transparency), public sector management (policy implementation, performance management), or financial management (budget execution, expenditure quality). Information is available, albeit to a variable degree of regularity and accuracy, that covers all the main bases of inputs, activities, outputs, outcomes, and poverty impact. The volume of data and analysis to support government operations has certainly improved over the period since Uganda initiated its Poverty Eradication Action Plan (PEAP), although there are still shortcomings in terms of scope, prioritisation, and data quality and use. PEAP/PRSP: The PEAP represents the point of departure for national M&E efforts. It is the national development planning framework that guides sector and district planning, as well as the budget process. It also serves as Uganda’s Poverty Reduction Strategy Paper (PRSP). The PEAP goals and objectives1 (annex A) have not been directly derived from, but are essentially congruent with, the Millennium Development Goals (MDGs). Its overarching goal is to reduce the headcount incidence of poverty in Uganda from 44 percent in 1997 to 10 percent by 2017. Every two years the government of Uganda (GoU) prepares the Poverty Status Report, outlining progress in reducing poverty and forming the basis for revision of the PEAP/PRSP. In addition, annual PRSP Progress Reports are produced, on the basis of the GoU’s Background to the Budget document. Poverty surveys and data collection: Uganda has a good demographic and household poverty dataset, which allows monitoring of national progress against the MDGs and other development indicators (see table 2.1). The major source of data is the regular schedule of surveys undertaken by the Uganda Bureau of Statistics (UBOS). Household income and expenditure surveys are conducted every two years, demographic and health surveys every five years, and censuses every decade. Moreover, UBOS maintains a district resource endowment profiles database, comprising information about, for example, topographical characteristics, natural resources, and land use in the districts. The Uganda Participatory Poverty Assessment Process (UPPAP), implemented by Oxfam, provides a qualitative dimension to analysis of poverty through bringing the perspective of the poor into planning.2 Client satisfaction with public services has been surveyed through a national service delivery survey, which was integrated within UBOS’s responsibilities in 2003. However, although UBOS has received capacity development assistance from a half-dozen donors and is able to produce a relatively good series of statistics, indications are that much of its data are not effectively utilized.

1. As refined by the Ministry of Finance, Planning and Economic Development (MOFPED) in the Poverty Monitoring and Evaluation Strategy, Kampala, 2002.

2. A first UPPAP was undertaken in 1999–2000 and a second in 2003.

2

Table 2.1. Schedule of Poverty Data Collection Survey Scope/coverage Frequency National Household Survey Income and expenditures, all districts 2-yearly Participatory Poverty Assessment Qualitative aspects of well-being, focus groups 2-yearly National Service Delivery Survey Satisfaction with public service delivery 2-yearly Demographic and Health Survey Health and demographics, all districts 5-yearly Census Demographics, population survey 10-yearly

Poverty Monitoring and Evaluation Strategy: Building on an informal Poverty Monitoring Network that has met since 2000 and comprises ministries, UBOS, Makerere University, nongovernmental organizations (NGOs), and donors, a Poverty Monitoring and Evaluation Strategy (PMES) has been formulated. The PMES is the product of a national effort, and it represents an overarching plan for M&E within the context of Uganda’s PEAP/PRSP. The PMES identifies a set of 33 priority indicators for implementation of the PEAP, for which a systematic effort has been made to establish a baseline and target (annex A). The PMES also addresses institutional responsibilities for tracking and review of poverty. Moreover, it seeks to draw the linkage between poverty indicators on the one hand and the planning and operations of ministries and their service delivery chains on the other. Poverty Monitoring and Analysis Unit: This unit in the Ministry of Finance, Planning and Economic Development (MOFPED) has played a particularly important role in the development of Uganda’s national M&E arrangements by virtue of its responsibility for the biennial Poverty Status Report, the establishment of the Poverty Monitoring Network, the formulation of the PMES, and its coordination of the UPPAP and other participatory poverty research in the country. The unit functions as an integral part of the ministry, although its staff is not part of the permanent government establishment. It has received support from various donors for specific tasks, and the United Kingdom’s Department for International Development has provided core funding support. Budget planning and expenditure controls: Uganda’s medium-term expenditure framework (MTEF) represents the interface between PEAP goals and operational planning and management. The budget is built through preparing sectoral and district level budget framework papers (BFPs) that spell out respective goals, achievements, and expenditure plans. A key element of reform has been the attempt to instill a greater degree of performance orientation by way of sectors being requested to relate their plans to the outputs and outcomes that follow from their spending. Training in results-oriented management has also pursued clarity of ministerial and departmental goals and expectations.3 A further component of reform has been tighter control of spending. Unexpected revenue shortfalls are matched by expenditure reductions. The introduction of a commitment control system prevents buildup of arrears. Other measures to control spending include the 2002 Public Finance Accountability Act which provides that supplementary expenditure must be approved by Parliament before the release of funds. To reduce deviations between budgeted and actual annual expenditures, the GoU publishes a biannual Budget Performance Report.

3. See, for example, Ministry of Public Service (2002).

3

Poverty Action Fund: To fund poverty priorities within the PEAP, the Poverty Action Fund (PAF) was created in 1998, comprising a combination of savings under HIPC debt write-off, general and earmarked donor budget support, and the GoU’s own resources. It provides a mechanism for ring-fencing expenditures that directly reduce poverty—primary health, primary education, water and sanitation, agricultural extension, and rural roads. Within Uganda’s system of decentralized management, local governments prepare activity-based annual and quarterly workplans and report separately for each of the 23 individual PAF and other conditional grant schemes. The PAF monitoring committee, which reviews policies and operations, meets quarterly and includes donors, line ministries, and NGOs. Coordinating role of sector working groups: Uganda’s sector working groups bring together key central ministries, the respective technical agencies, NGOs, and the donor community, and they function as coordinating bodies for sector-wide strategy planning, budget preparation, stakeholder consultations, and liaison. Their biannual meetings include a comprehensive assessment of sector-wide progress. Most GoU ministries have research and documentation departments with skills and facilities for fairly advanced data collection. Several ministries have established their own management information systems (MISs). Following the decentralization of service delivery, their focus is on reporting and compiling data on sectoral PAF programs for which local authorities have responsibility. Civil society’s monitoring: Civil society organizations (CSOs) are active participants in the national budget process, including the sector working groups. The Uganda Debt Network coordinates civil society monitoring of PAF activity in the districts. An independent NGO often critical of government and the donor community, it is also piloting a community-based M&E system, which mobilizes the public to verify expenditures and service delivery. A client scorecard system, to be piloted in the health sector, is currently under preparation. A particularly noteworthy public monitoring practice in Uganda is the posting of notices of funding allocations, which has been made mandatory for all PAF workplans and funds (MOFPED 2001), forcing transparency and accountability into administrators’ dialogue with clients or end users. This policy was introduced after the results of the first (1996) public expenditure tracking survey (PETS) showed a significant degree of “leakage” of government spending in the education sector.4 Evaluations, studies and reviews: Uganda’s government ministries, research institutions, and donor organizations are continuously undertaking a number of ad hoc poverty studies and reviews. At the sectoral level such studies and reviews are increasingly being undertaken under the auspices of the sector working groups. The Economic Policy and Research Centre and the Makerere Institute of Social Research, in particular, have strong capacities for research and evaluation. With support from the World Bank, the first PETS was conducted in the education sector in 1996. Since then expenditure tracking studies have been undertaken in the health, education, and water and sanitation sectors. However, the planning and conduct of such studies is not always well coordinated, and the PMES envisages a consolidated process of agenda setting for ad hoc poverty research and reviews.

4 See, for example, Reinikka and Svensson (2002).

4

Accountability and transparency: Uganda has a well-developed infrastructure of accountability institutions at the central, sector, and local government levels (Table 2.2). A national anticorruption strategy was announced in 2001 (Directorate of Ethics and Integrity 2001). The plan aims at building capacity for coordination, planning, and monitoring anticorruption efforts. In this connection GoU is reviewing all anticorruption laws to consolidate them into a single comprehensive legislation. The 2002 Leadership Code Act requires that senior government officials (ministers, presidential advisers, permanent secretaries, directors, and senior project managers and coordinators) declare their income, assets, and liabilities. Accountability institutions are partially funded by the PAF, under which 5 percent of resources have been allocated to monitoring and accountability. Nevertheless, and in spite of also being augmented by selective external assistance efforts, the accountability institutions suffer from continuing resource and staffing constraints.

Table 2.2 Inspection and Audit Agencies Inspection Audit Central Office of the Prime Minister (OPM)

Directorate of Ethics and Integrity Inspectorate General of Government MOFPED Treasury Inspectorate Ministry of Local Government Inspectorate

Public Accounts Committee of Parliament Office of the Auditor General MOFPED Office of Accounts and Internal Audit External donor audits

Sector Sector ministry inspectorates; Sector ministry internal audit District District inspectorates Local public accounts committees

District internal audit

External support for M&E skills and training: There have been many M&E capacity development efforts attached to individual donor-funded projects. More generic M&E capacity development support has also been provided to individual ministries and agencies. Examples include UBOS, the Auditor General, and various departments of Makerere University. These bodies have received external assistance from a range of donors, including Denmark, Norway, the United Kingdom, the United States, the World Bank, the UNDP, the UNPF, and the UNCF. With support from the World Bank’s OED, a number of senior Ugandan civil servants have participated in international M&E seminars and training courses. With support from OED, about two dozen officials have taken part in OED conferences, the International Program for Development Evaluation Training (IPDET) at Carleton University, Canada, and the two conferences convened by the African Evaluation Association (AfrEA). Skills gained and networking through these events were instrumental in the establishment of the Uganda Evaluation Association (UEA) in 2002. Within the PRSO context, a series of M&E seminars, workshops, and consultations has been held with senior officials from Uganda’s central and sector ministries, as well as CSOs and donors. Building on OED diagnostic guidance material,5 donor analytical work and M&E advocacy under the PRSO have been captured by two separate studies: an initial M&E diagnosis in 2000–01 (Hauge 2001)6 and a more detailed study of M&E in the health, education, and water sectors in 2002 (Hauge and others 2002). Although both these studies have presented a critique of Uganda’s national M&E arrangements, their presentation was well received by government officials. An overview of OED’s involvement in M&E capacity development in Uganda is presented in annex C.

5

3. WHAT CONTRIBUTION IS M&E MAKING TO NATIONAL DEVELOPMENT?

In terms of the desired outcome of M&E capacity development7that M&E is effectively used to improve the quality of decision-makingsome progress is being made in Uganda, although it is inherently difficult to identify clear evidence of this. Attention to M&E, even if sometimes only arising as an implicit concernfor example, in the context of broader public management reformsis one of the factors bringing greater rationality and discipline to Uganda’s public finances and development management. M&E is providing the GoU with a more evidence-based foundation for policy formulation, budgeting, and operational management. The increased production and availability of information concerning national poverty eradication efforts is also allowing civil society and Parliament to assume a more meaningful role in public affairs. Improved economic management: The M&E that has accompanied Uganda’s budget reforms has helped rationalization of public finances and achievement of sustained macroeconomic stability. First it has introduced a planning horizon that extends beyond individual projects and financial years—that is, it is linked to the medium- and longer-term national development goals of the PEAP. Second, through building on a hard revenue constraint and expenditure monitoring, sector planning has been infused with realism and discipline, leading to relatively low levels of inflation and interest rates. Third, the establishment of the PAF has made it possible to increase the relative priority given to activities that have a direct bearing on poverty. The share of PAF expenditures within the national budget has risen sharply, from 18 percent to 37 percent between 1997–98 and 2002–03, resulting in a major reorientation of national expenditure. Participation in public decision-making: The process of national decision-making in Uganda, in particular at the central government level, is generally characterized by relative transparency and openness. For instance, a number of measures to keep the public informed and involved throughout the budget cycle have been introduced (Gariyo 2000, MOFPED 2000). All key documents are publicly available, often at the draft stage. In addition, since 2001 a vigorous effort has been mounted to increase the effective participation of the legislature in the budget process. The 2001 Budget Act allows for the cabinet and the legislature to receive BFPs early enough to provide detailed comments before the budget is finalized. A result of opening the budget process and the greater availability of data on government operations has been that civil society increasingly plays a role in both the planning and review of national- and sector-level projects and programs. A wide cross-section of CSOs takes part in sector working group meetings that prepare BFPs. Decline in transactional costs of donor liaison: Both the value and the number of stand-alone, donor-supported technical assistance projects in Uganda have recently declined. Between 1998 and 2002 the number of health, education, and water projects has been reduced by more than half, from 111 to 54, while the value of annual disbursements has declined from nearly US$150 million to approximately US$25 million. The planning and management of projects that remain are increasingly being undertaken with reference to the PEAP framework and the respective sector-wide approaches. An increasing share of external assistance to Uganda, currently about 60 percent, is provided as budget support. A growing number of other donors, including Ireland,

6

Germany, the Netherlands, the United Kingdom, and the African Development Bank, provide parallel financing to the PRSO. With the shift toward programmatic assistance, alignment with national objectives is ensured and the transacational costs of liaison reducedand the GoU is mindful that donors would only substitute their own M&E systems for those of the government once they have faith that national arrangements maintain meaningful standards of accountability and fiduciary control. Despite the move to budget support, there are still 42 donors who in total have 524 active projects and who have entered into 825 separate agreements with the GoU. It is clear that even the remaining ad hoc, off-budget donor projects place a severe M&E burden on the GoU officials involved. Cost-effectiveness is still elusive: Although substantial gains in poverty reduction have been madefor example, the proportion of people living below the national absolute poverty line decreased from 56 percent to 35 percent between 1991–92 and 1999–2000progress does not appear to match the rate of increase in budget resources for the social sectors.8 It is not evident that the observed poverty reduction can be linked to any improvement in the quality of public sector management. There are indications of continuing poor effectiveness and value for money in public service delivery. For example, although the total flow of resources to the water sector tripled between 1997–98 and 2000–01, the annual number of new water points dropped by 8 percent. Moreover, between 1995 and 2000 there was a reversal in infant and under-five mortality as well as immunization coverage. M&E as a possible constraint to effectiveness: A concern with M&E may translate into a call for expansion of M&E activity. However, the key challenge faced by Uganda is not more, but better, M&E. Although many useful M&E instruments and practices have been introduced, the M&E arrangements in place in Uganda have also in some ways constrained effectiveness in making progress with the PEAP:

• Disparate M&E information flows create confusion about goals and complicate policy assessment and budget analysis.

• Parallel sets of M&E arrangements lead to duplication and waste in data collection. Underused data discourage efforts to improve data quality.

• Excessive workload of data collection, inspection, and reporting can divert managerial attention from productive service delivery.

• When incentives are tied to conduct of M&E, as opposed to the underlying performance revealed by M&E, managers are rewarded for good paperwork rather than contribution to poverty eradication.

In Uganda, there are examples of M&E “harmonization” initiatives that have added to, rather than simplified, M&E. New systems, even if logically sound, are unlikely to constitute any improvement unless they are also accompanied by simultaneous reductions in M&E elsewhere. In situations with several parallel information collection and reporting systems, data redundancy, duplication, and waste become more likely. Familiarity with the numerous guidelines and reporting formats becomes a major workload. Focus on downstream results is dissipated, and capacity development is undermined. Information that flows from different M&E streams cannot easily be aggregated or compared for purposes of broader, cross-sectoral policy and budget analysis, including for preparation of the PEAP/PRSP.

7

The objective of national M&E systematically yielding quality data to buttress a public management system oriented toward performance, cost-effectiveness, and results of service delivery remains an important challenge for Uganda. If synergy and simplification is brought to the set of M&E systems and practices that have been put in place in Uganda, it will constitute an important step in the right direction. Many senior government decision-makers and other key local stakeholders have understood that good M&E is important for the attainment of national development goals. In particular the PEAP and the PMES show evidence of a widening appreciation that sound M&E is not merely a data collection issue, but is also intrinsic to results-oriented public management practice. The increased interest in and attention given to M&E in Uganda emanates from a combination of domestic and external sources of demand, founded on common concerns about performance and results orientation in public service delivery. If M&E has made a positive contribution to improving public management, it has often been more from its implicit importance—as part of wider reform efforts—than from technical M&E development efforts. An emerging understanding of M&E’s role has helped “raise the bar” for assessment of GoU poverty eradication efforts. PEAP revision and PRSP progress reporting leave an impression of coherence and substance in analysis that is often lacking in other PRSP countries. At the sector level attention is shifting, albeit slowly, from the monitoring of ministry activities toward greater attention to service delivery targeting and outcomes. Cost-effectiveness questions are being raised and discussed, even if not yet resolved. Growing transparency and participation bring a degree of assurance that the GoU’s actions are congruent with citizens’ own perceptions of need.

8

4. WHAT CHALLENGES REMAIN? Systemic M&E challenges: In terms of results-oriented M&E, four sets of problems remain:

1. The “missing middle”: Uganda’s national M&E systems produce relatively accurate measures of expenditures and activities on the one hand and poverty or impact on the other, but there are relatively few systematic data available at the middle level of outputs, reach (the coverage of services and other outputs), and outcome. (This is a common problem in PRSP countries (Booth and Lucas 2001). Service providers, for instance, rarely collect client satisfaction data.

2. Link between stages of results chain: Within Uganda’s public sector, existing M&E

systems are generally not geared toward understanding causality and attribution between the stages of development change. The evaluation function is relatively undeveloped. Although tracing causality through the results chain always involves methodological challenges, an understanding of the processes of producing results from government and donor spending is, nevertheless, at the heart of the performance concept, whether in terms of contribution to outputs, outcomes, or impact.

3. Feedback and use of M&E data: In Uganda incentives for civil servants are often tied

to nominal compliance with reporting requirements, rather than responding to the underlying performance revealed by M&E. More broadly the demand for evidence-based decision-making is not always present. Poor performance and misconduct, for instance, are rarely sanctioned. Also, little feedback is provided on data collected through ministerial inspection. What matters with M&E is not so much the data that is collected or the facts that are available, but how the data are used to inform choices in the different stages of planning and public service delivery.

4. Multiple, parallel data collection systems: In Uganda planners and managers at the

sector, district, and facility levels currently have to relate separately to the numerous different M&E arrangements that have been established for the different conditional grants and national funding schemes, as well as the different donor requirements that remain. The result is a complex and formidable burden of inspection activity, indicator data collection, and reporting formats.

These problems have, in turn, led to poor-quality M&E data in terms of missing, inaccurate, or outdated information. The distinction between observed reality and what is hoped for is often blurred. Although the M&E systems and practices that are in place arguably provide a reasonable accountability framework, their contribution to substantive learning is more limited. Uganda’s M&E system produces a large volume of data on compliance with rules and regulation, but it is often of poor quality or not fully used and does not yield a sufficiently clear basis for assessment of value for money and cost-effectiveness in public service delivery. Poor data quality and lack of demand for performance information are mutually reinforcing in undermining efficacy of M&E systems.

9

Making further headway with M&E will necessarily be an ongoing and long-term process of awareness building, institutional liaison, systems adjustment, and skills formation, which will require coordination and operational action on several fronts. Further reforms will be necessary at the individual sector or ministry level, but cross-sectoral coordination will also be needed to bring coherence, focus, and synergy to Uganda’s M&E systems. The PRSO as entry point for M&E reform: The PRSO supports national M&E capacity development in two main ways. First the PRSO emphasizes close liaison between donors and the GoU through budget support, thus inducing more reliance on the government’s own M&E systems and practices and less emphasis on the multiple and disparate donor M&E rules and requirements. Second M&E is identified under the PRSO as an area of cross-cutting governance reform, together with changes in the related areas of pay reform, budget management, accountability, and anti-corruption, and engagement with civil society. The focus of M&E work is the PRSO policy matrix, which represents a “rolling” plan of issues to be addressed and specific actions needed for national M&E systems and practices to bring forth data that are of high quality, are fed into decision-making, and can bring a focus to cost-effectiveness and value for money issues.9 In this respect the PRSO constitutes the national M&E reform agenda. To coordinate the review work and to follow up on implementation of the PRSO M&E benchmarks, an M&E working group has been formed, under the leadership of the Office of the Prime Minister (OPM) and comprising MOFPED, and the Ministries of Public Service and Local Government. M&E responsibilities and coordination arrangements: The Ugandan government has realized that stronger coordination is needed to ensure that M&E helps guide public actions toward greater cost-effectiveness in pursuit of poverty eradication. Coordination is needed to improve information standards, requirements, and systems that support different levels of national decision-making. Because assignment of responsibility for overall M&E coordination can ultimately only be resolved from the political level, the M&E working group is preparing a strategy for national M&E coordination, which has recently been submitted for review by the Cabinet. As a starting point for establishment of an integrated national M&E system, the GoU has reached agreement on a structure of responsibilities for implementation of cross-cutting public management reforms under the PRSO as a whole. At the same time the National Planning Authority, with a possible role in M&E coordination, has recently been established. Building on a clear and agreed set of M&E responsibilities, the development of systems and skills also needs to be subject to coordination. In this respect the World Bank’s Uganda Capacity and Performance Enhancement Project (CAPEP), a US$100 million facility currently under preparation, may represent a useful entry point. Clarity of targets for outputs and outcomes: Over the last few years, significant improvements have been made to sector-level goals and targets, as reflected in budget requests, sector strategies, and policies. However, much still remains to be done in terms of the results orientation and consistency of sector planning and performance management. Specifically, several ministries have done their planning on the basis of measures that essentially relate to inputs, activities, and processesthat is, with a focus on the spending side, as distinct from the outputs and outcomes that are meant to result. Nevertheless, improvements have been made in, for example, the health, education, and water sectors by identifying output and outcome targets in the budget cycle. The GoU has similarly committed to a special effort to identify output and outcome targets for the

10

agriculture and the justice, law, and order sectors. For the 2003 Public Expenditure Review, the GoU and its partners are evaluating the results orientation of the overall budget and, specifically, the progress made in five sectors: health; education; water and sanitation; agriculture; and justice, law, and order. The currently ongoing second revision of the PEAP is also expected to lead to an improvement in the clarity and results orientation of sector goals and targets. Common district-level data collection: Currently the various line ministries in Uganda have separate and parallel systems and routines for compiling and reporting data from districts and local governments. As reported in annex E, there is considerable overlap between these systems, and much of the data collected appear to be either redundant or of poor quality. The combined workload is in any case a substantial burden on district planners and sector program coordinators. There are also definitional differences in basic concepts and terminology of M&E under the various M&E systems. Until recently the data burden on districts and ministries had been expected to increase as a result of a new data system planned by the Ministry of Local Government under the Local Government Development Programme (LGDP). However, under the aegis of PRSO liaison, it has been agreed that the GoU will instead seek to build a common platform for local-level information collection and data entry, where a single system will serve the most essential information needs of both the central and the line ministries. Consolidation of inspection: Uganda’s districts and frontline service delivery units are burdened by excessive visits by supervision, support, and inspection teams from various ministries and agencies—these teams often have different objectives but frequently ask for similar information. Those visited perceive inspection as unnecessary policing, rather than constructive support or quality assurance, and also observe that it reduces the time available for actual service delivery. There is poor feedback from inspection visits, and little of the data collected are used for any decision-making at the local level. Inspectors are believed to be often motivated by per diem remuneration for travel and to give little attention to service facility client concerns, and they are rarely able to address value-for-money issues. Most of the inspection activity been conceived in terms of people from the centre making physical visits to the district or facility, with an objective of verifying the content of district reports. A very large workload is involved in catering to inspectors. In the health sector alone, this potentially involves as much as the equivalent of 1,400 staff-years annually. There are opportunities for reducing both the inspection activity that takes place within ministries and the inspection undertaken by the different national accountability institutions. One option would be to consolidate inspection activity into coordinated multi-disciplinary teams of technical ministry personnel and central accountability representatives. There are also a number of inspection tasks that can be undertaken by CSOs or communities themselves, such as verification of classroom or health clinic construction. Agenda setting for conduct of evaluations, ad hoc studies, and reviews: Evaluation is not systematically embedded in the GoU’s management practices. Most evaluations have been instigated by donors. The lack of coordination of studies and reviews has led to either overlap between different research exercises or reports of such work being poorly fed into decision-making. As part of the PRSO policy matrix, and as envisaged by the PMES, the GoU intends to establish a mechanism for agenda setting for evaluations, ad hoc studies, and reviews. Because evaluation addresses issues such as actual progress in attainment of program objectives, cost-effectiveness and value for money, it responds to some of the aspects of Uganda’s M&E system that are most critically lacking. Expenditure tracking studies in the Uganda health and education

11

sectors have brought about a better understanding of various budget execution problems by identifying substantial problems with “leakage,” delays in flow of funds, and capacity problems related to financial and operational management. To date, the expenditure tracking studies have largely been instigated and funded by donors, although their usefulness has since been attested to by the GoU itself. Other sectors also need to do tracking studies to improve flow of funds and enable efficient use of limited resources. Use of public service client satisfaction data: To rectify the lack of systematic information on client satisfaction with public services, a national service delivery survey (NSDS) was conducted in 2000 and will be repeated in 2003. Client satisfaction and public perceptions of service delivery quality are important issues with which civil service managers at both the facility and ministry levels should be concerned. The findings of the NSDS can be adopted as a barometer of government service reach and used for comparing performance at different locations and over time, for gauging the targeting effectiveness of government service delivery, and for performance target setting. More generally, changes in the level of user satisfaction can indicate to managers where they need to make changes, and levels of client satisfaction can provide the yardsticks of performance for which managers can be held accountable. A further, specific potential use of the NSDS would be the articulation of “client service charters”commitments for facility-level and ministry performance. However, the first NSDS has not been well used as a basis for performance improvement. One reason is that the surveyed coverage did not correspond well with functional areas of service delivery responsibility. Strengthening public accountability: Although popular dissatisfaction with Uganda’s public service delivery remains widespread, many communities still feel that they are not empowered to question their leaders or service delivery managers. Bottom-up accountability is embedded in Uganda’s Constitution and Local Government Act and is an aspiration in sector policies, but putting in place actual instruments of accountability has been weak. At the local level people still do not feel involved in decision-making.5 The mandatory posting of notices on PAF allocations has been patchily practiced, presumably in part because there are no sanctions for civil servants who fail to comply with this requirement. The GoU intends to review how direct, bottom-up public accountability can effectively complement the established internal accountability mechanisms of government itself. An initiative is under way to establish a system of facility-level client scorecards to be managed by CSOs. Uganda’s Parliament has recently assumed a strengthened role in the government budget process but is still considered relatively weak compared with the civil service. The UEA is emerging as a professional body with a capacity for contributing to the independent assessment of government actions and performance. The challenges that remain will need to be addressed at several levels. Individual institutions and ministries will have to take the lead in, for example, clarifying their goals for contributing to national poverty eradication and measuring client satisfaction with their service delivery. At the same time central coordination will be needed to ensure common approaches to such tasks and identify overlaps and potential synergies among different planning, data collection, and reporting systems.

5 See, for example MOFPED (2003) and Brock, McGee and Ssewakiryanga (2002).

12

5. LESSONS LEARNED FROM UGANDA’S M&E EXPERIENCE

The final M&E goala material improvement in effectiveness of the public sector in alleviating povertyhas not yet been reached in Uganda. However, there has been important progress in many areas, and a number of lessons emerge that are noteworthy for Uganda and other PRSP countries. Perhaps most important, a domestic demand is slowly emerging for M&E arrangements that stimulate improvements in data quality and bring cost-effectiveness and value for money to public service delivery. Leaving national authorities in the driver’s seat: A key lesson from Uganda’s M&E experience is that the aspects of M&E that work best are those designed for and by national decision-makers. Uganda’s PEAP and PMES are genuinely home-grown products that emerged wholly from within national institutions. The practice of issuing public funding notices was similarly implemented without external support. Moreover, sector working groups have emerged in accordance with the decision-making and stakeholder consultation habits and practices that prevail in the country. This resonates well with the lessons learned from a generation of evaluating development assistance: capacity development needs to be founded on the national decision-making realities of each country. Conversely, if M&E were to be built primarily around the needs of the external donor community, the risk would be that skills and resources would be drained away from the domestic core structures of government. And if M&E were to be organized as enclaves operating according to rules and regulations that are separate from those that apply to national authorities at large, indigenous capacities would be undermined. M&E capacity development needs to address policy level demand: If M&E is undertaken only as a technical skills and facilities exercise, it can add impetus to the proliferation of systems, routines, formats, and requirements, which in turn lead to an unnecessary workload and accumulation of redundant and often poor-quality data. It is better to have a small amount of relevant and reliable data than large volumes of data that address peripheral questions and are of questionable quality. However, improvements in data quality will emerge only if M&E capacity development is tied to the broader policy-level environment of demand for results orientation and value for money. M&E must emphasize the attainment and cost-effectiveness of poverty-reduction results that emanate from public action, instead of merely providing documentation of activities and compliance with rules and regulations. CAPEP will help to further strengthen M&E skills and can be expected to raise both awareness of and demand for M&E within the civil service. However, in Uganda as in other countries, the biggest constraints to M&E capacity and use are in the policy and institutional environment that surrounds the M&E functionin particular the structure of incentives, decision-making practices, and public service management culture. Ultimately M&E will flourish only where there is a policy-level demand for what it produces, where managers seek an empirical and factual foundation for decision-making, where the practice of M&E follows as a consequence of the incentives embedded in public service systems, where rewards and sanctions are guided by achievement of results, and where managers collectively recognize a self-interest in adopting tools of continuous assessment and learning.

13

Evaluation is a necessary complement to routine monitoring: Monitoring appears most critical in terms of daily operational functioning, but routine data collection, even if accurate and timely, can rarely produce a satisfactory picture of what contribution public service delivery actually has made to poverty eradication goals. In contrast, evaluation is a more reflective functiona periodic exercise aimed at understanding the interrelationship between inputs, activities, and outputs on the one hand and outcomes on the other. Both monitoring and evaluation are needed to make sure there is an effective process of transformation through service delivery results chains, from inputs and activities through outputs to outcomes and, ultimately, impact. Use of evaluation is an integral component of sound management practiceof equal importance at the project, program, strategy, and policy levels. There is a growing awareness in Uganda of the importance of evaluation as a function to complement more routine monitoring. As reported by the PMES, evaluation is no longer regarded as a somewhat bothersome and esoteric bureaucratic requirement that comes with external donor projects. A national evaluation association has been formed, and Uganda has been an active participant in the nascent AfrEA. Civil society can do some M&E functions better and cheaper than government: Results orientation improves when the government’s internal M&E systems are reinforced by the public monitoring and reporting on how public service providers perform. Few public service users are able to judge complex engineering matters, but they are experts on whether public services meet their needs and expectations, whether specific aspects are satisfactory or unsatisfactory, and whether government service providers at the facility and agency levels are responsive, reliable, and accountable. End-users’ perceptions of public service delivery derive from the issues of whether services are wanted, useful, and effectively delivered. Direct feedback from public service users can be an input to both frontline and central decision-making. The Uganda experience with public notices of PAF allocations has shown how transparency can have a direct, positive effect on service delivery. Managers become more responsive to the needs and satisfaction of their client constituency. There are also M&E functions that civil society can perform at a lower cost than government. Groups such as parent-teacher associations can inspect basic school facilities, for instance, without costs of travel and per diem allowances. However, bottom-up accountability and transparency work only when evidence on performance also has consequences. Where transparency brings malpractice to the surface, sanctions should be applied to those who have transgressed. However, where facilities display high levels of client satisfaction, it should count toward the rewards to managers. Unclear goals and objectives can frustrate a results orientation: When different planning mechanisms, management arrangements, and M&E initiatives entail inconsistent objectives, there is a risk that managers will become confused as to what the “real” or highest-priority goals are. Any one set of goals can be perceived as nominal in value, with perfunctory reporting obligations or, even worse, opportunities for “gaming” the numbers. The multiplicity of goals and indicators in Uganda is shown in annex E. There has recently been movement in Uganda toward convergence and outcome orientation of performance and planning indicators. The sectors are becoming clearer about the results they wish to create—in the education sector, for instance, goals are now being set for school completion and educational attainment. At the same time the PMES has refined a set of priority indicators that are more closely focused on poverty outcomes and more systematic in ascertaining availability of data on the baseline and target.

14

Poverty-eradication goals must also be linked to managerial use: In addition to clear goals for impacts on poverty, the daily operations of any ministry, department, or agency will always require M&E of inputs, activities, and outputs that cover resources, personnel management, organizational development, and so on. Managers cannot manage with data about poverty alone. However, if M&E is exclusively tied to process issues, the risk is that managerial attention will slip away from the goals that are being pursued. Left to their own devices, government agencies will generally establish performance measures and M&E arrangements that relate to the efforts and activities over which they have a high degree of control instead of the harder-to-measure results that follow from what they do. Outcomes can come to be seen as someone else’s business. However, understanding poverty should be the point of departure for planning resource use and activities, rather than the other way around—continuing with business-as-usual on the assumption that useful outcomes will somehow appear. Poverty trends need to be understood in terms of the service delivery process. Strategies that are relatively more effective or economical than others need to be identified and encouraged. Although it can be difficult to attribute changes in poverty outcomes to individual agencies with much certainty, the vantage point of seeking to understand contributions to real change needs to be adopted as a continuous focus of managerial attention. What matters with M&E is not so much the data that are collected or the facts that are available, but how it is used to inform choices. Therefore M&E needs to be designed with a clear view of how it will feed into decision-making. M&E needs to get to the right people at the right time. In Uganda the process of bringing managerial attention to poverty outcomes is an ongoing exercise in which important progress has been made. However, much unfinished business remains, and M&E will continue to play a critical role.

15

6. CONCLUSIONS Attention to M&E, even if arising as an implicit concern with wider public management reforms, has been an important factor in the improvements that have been observed in Uganda’s public management over the last few years. A number of M&E-related initiatives have helped instil an orientation toward the poverty-eradication goals of the PEAP/PRSP. Both policy formulation and economic management have been aided by a more comprehensive availability of information. However, although systems are in place for expenditure controls and tracking poverty trends, current M&E arrangements have not yet brought a sufficient focus to client orientation and cost-effectiveness issues in public service delivery. Within the government many incentives are tied to compliance with reporting requirements for their own sake. A large amount of data is collected, but they are often of poor quality and are not effectively used to inform decision-making. The Uganda experience dispels the notion that increased M&E, in and of its own, will lead to improvements in results orientation. What Uganda needs is not more, but better, M&E. Rather than volume of M&E activity, what matters is the quality of M&E systems and practices in funnelling managers’ attention toward a clear and coherent understanding of what difference they do or can make to national development. Moreover, if not designed to tie in with decision-making, M&E systems can undermine results orientation with time wasted on mere accumulation of data and the risk of goal confusion. Likewise, data quality problems cannot be resolved purely by technical interventions at the data collection level, but must be addressed in terms of the environment of incentives, demand, and information use. It must be recognized that donor support for M&E capacity development, when tied to their own individual accountability and reporting requirements, adds to the proliferation of systems and practices. In this particular respect donors have been part of the M&E problem rather than the solution. What recipient countries ultimately need, for the sake of sustainability, is not a capacity for adhering to the M&E requirements that accompany donor support, but a capacity for managing without these donor requirements. Making further headway with M&E in Uganda is largely a public governance issue, rooted in policy-level demand rather than shortcomings of technical systems for information management. Many of Uganda’s national policy-makers are aware of the systemic role that good M&E plays and can continue to play in attaining success with Uganda’s poverty-eradication goals, as well as the challenges involved in reforming long-established public management practices. The PRSO process has helped bring M&E to the agenda of cross-cutting reforms. As a programmatic lending instrument it has, at the same time, brought a degree of comfort to donors’ reliance on the GoU’s own M&E systems and practices. The challenge for Uganda is now to bring coherence and synergy to its national M&E arrangements. In this regard the role of cross-sectoral coordination is being addressed by the OPM. This represents an important first step, but still leaves other, often difficult, reforms to be addressed.

16

ANNEX A. Poverty Monitoring and Evaluation Strategy: : Priority Poverty and Sector

Indicators PEAP pillar & PMES indicator Frequency

of reporting

Target (by year)

Current status (year)

Pillar 1. Economic growth and transformation • Gross domestic product (GDP) growth rate • % of national budget used for poverty focused programmes • Inflation rate • Domestic Revenue/GDP • Foreign exchange reserves

Annual Annual Annual Annual Annual

7%

5%

5 months

5.6% (01–02) 35.4% (01–02) 4.5% (2001) 11.7% (2001) 4.4 (2001)

Pillar 2. Good governance and security • Incidence of misappropriation of public funds at national, district level • Number of people internally displaced, by sex, age, and location • Beneficiary assessment of quality of service (police and judiciary) • Level of awareness about rights, entitlements

Annual Annual Biannual Annual

0%

None Good High

610,000(2000) Very poor Very low

Pillar 3. Increasing incomes of the poor • Economic dependency • Poverty indicators—incidence, depth • Share of rural non-farm employment, by sex and location • Yield rates of major crops • Proportion of land covered by forest • GDP per unit of energy use

Biannual Biannual Biannual Biannual Annual Annual

10%

400%

35% (2000)

Pillar 4. Improving quality of life • Life expectancy in years, by sex 5 years 43 (2000) • Infant mortality per 1,000 live births 5 years 68 (05) 88 (2000) • Maternal mortality per 100,000 5 years 354 (05) 505 (2000) • Nutrition (stunted) 5 years 28% (05) 39% (2000)

(a) Health • Immunization coverage (DPT3) Annual 60% (05) 46% (2000) • Percentage of approved posts filled with qualified health workers in

public and private-not-for profit facilities Annual 50% (05) 40% (2000)

• Deliveries in public and PNFP facilities Annual 35% (05) 25% (2002) • Human immunodeficiency virus (HIV) prevalence Annual 5% (05) 6.1%(2001)

(b) Education • Literacy rate, by sex and location Biannual 50% (07) 63% (2001) • Net school enrollment, by sex and location Annual 98% (03) • Pupil to trained teacher ratio Annual 50:1 (03) 54:1 (2001) • Pupil to textbook ratio Annual 6:1 (03) 4:1 (2001) • Pupil to classroom ratio, by location Annual 92:1 (03) 98:1 (2001)

(c) Water and Sanitation • Number and % of population within 1½ km to safe water Biannual 60% (04) 52% (2001) • Number and % of population with good sanitation facilities Biannual 60% (04) 50%(2001)

Source: MOFPED–Poverty Monitoring Network 2002.

Progress with poverty reduction: Since 1992 Uganda has made progress with poverty reduction at a faster rate than needed to attain MDGs by 2015.11 Economic growth has averaged 6.3 percent per year over the last decade, resulting in an annual 3.3 percent increase in GDP per capita. The proportion of people living below the national absolute poverty line decreased from 56 percent to 35 percent between 1991–92 and 1999–2000. Uganda has also posted gains in a number of broader social indicators, such as literacy, access to health care, and safe water. Since the policy of universal primary education was introduced in 1997, nominal school enrolment almost tripled, from 2.6 million in 1996 to 6.9 million in 2001. HIV/AIDS prevalence rates dropped from 30 percent in 1992 to 8.3 percent in 2000.

17

ANNEX B. Summary of OED M&E Diagnosis Findings, 2001

Summary of strategic issues, challenges, and possible actions Overall M&E task

Monitor and provide feedback on progress in poverty alleviation

Strategic M&E issues

1. Coordination and harmonization

2. Development goals, targets, and performance indicators

3. Incentives for contribution to results

4. Devolution of managerial autonomy

5. Role of civil society in enforcing transparency and accountability

6. PEAP partnerships principles

7. M&E skills training

Positive elements in Uganda

* Draft Poverty Monitoring Strategy * Sector Working Groups (SWG) as nexus of planning, budgeting * Efforts to harmonize project progress reporting * Earmarking of 5% of PAF funds for monitoring and accountability

* PEAP/PRSP as overall framework of poverty priorities * Training in ROM (results-oriented management) being piloted * “Indicator retreat” as part of budget cycle

* Recognition of service delivery effectiveness as imperative of public management * National Service Delivery Survey 2000

* Decentralized responsibility for service delivery * Introduction of output-oriented budgeting * Comprehensive district plans * LGDP capacity development

* Consultative nature of PEAP process * Transparency of budget process * Practice of public notices * Significant capacity of NGOs * Government–civil society dialogue at central level

* Draft CDF partnership principles * Trend towards budget support * Consultative group meeting scheduled as part of budget cycle

* Awareness of M&E importance * Availability of local researchers and local academic and training traditions

Some challenges being en-countered

* Separate planning and reporting formats for different funding sources * Sector/district policies, budgets, workplans approached as separate exercises * Alignment and coordination of different results management initiatives * 1/3 of ODA is TA outside of government budget

* Inconsistency in clarity of goals at sector level * PEAP goals correspond to ministerial activities rather than poverty outcomes * Few goals are defined with measurable timeframe, baseline, and targets * Weak linkage between sectoral and district goals

* Performance assessed in terms of expenditure and bureaucratic activity * Weak linkage between resource allocation and performance * Rewards geared to good paper-work * Inconsistent enforcement of sanctions for poor performance * Corruption largely unpunished

* Generally weak management capacities at local levels * Prescribed spending ratios of conditional grant scheme gives little flexibility for managers to adapt to local needs * Number and level of posts directed from the center

* Need for improved stakeholder consultation in priority setting * 1/3 of ODA is TA outside of government budgets and M&E practices * Gou/NGO dialogue at center is not mirrored at local levels

* 1/3 of ODA is TA outside of government budget and M&E * Nearly 300 stand-alone projects remain * 20 separate annual program reviews * Local donors cannot depart from corporate M&E guidelines

* Weak management skills at local government levels * Likely increase in demand for management and conduct of M&E dealing with interrelationship between service delivery and poverty outcomes

Possible actions to address challenges

* Identification of an M&E champion ministry or agency * Establishment of core M&E arrangements (such as via a formal M&E framework), harmonization of terminology, reporting formats and periodicity * Improved coordination between inspection and audit agencies

* Cascading of PEAP goals and targets through planning, budgeting, and work planning at sector, district, and facility levels * Long-term expenditure framework focus on defining medium-term PEAP goals and targets

* Introduction of reach and outcomes as yardsticks of success and performance reward* Use of the NSDS as barometer of client satisfaction improvements * Introduce value for money concerns in Finance Act *Introduction of client service charters

* Allow greater local autonomy over recruitment, salaries, and non-wage expenditures *Ensure stronger local oversight as the quid pro quo *Introduction of participatory M&E practices as key management function

* Extend transparency practice from allocation to execution * Client report cards as complement to the NSDS * Make NGOs eligible for PAF monitoring and accountability funding *Introduce client service charters

* Leverage of donor support for the CDF and PEAP to increase synergy in planning, reporting, and review * PRSO policy matrix as joint planning and review mechanism

* Strengthen local capacity for program evaluation skills training, for example, via training-of-trainers at national institutions * Coordinated use of funding earmarked for M&E under PAF, the LGDP, and Economic and Financial Management Programme II * Establishment of national evaluation association

18

ANNEX C: OED’s Contribution to M&E Capacity Development (ECD) in Uganda As part of its formal mandate OED has responsibility for encouraging and assisting developing countries to build effective M&E capacities and systems. OED plays a catalytic role, including support to a small number of countries with the intention that they will develop good-practice M&E approaches that can be adapted by other countries to suit their particular circumstances. This annex was prepared as an input to a self-evaluation that OED is conducting of its M&E capacity development (ECD) work. OED has provided active support for Uganda since 2000. A brief summary and timeline of this support is shown in table C.1, together with a number of subsequent M&E developments observed in the country. These are not exclusively attributable to OED efforts, but they represent results that OED’s work has contributed to, together with national stakeholders and other external partners. M&E outputs represent positive steps in terms of making national M&E more systematic and coherent. National M&E outcomes are broader improvements in national decision-making to which those outputs have contributed.

Table C.1. Timeline and Results Chain for UgandaM&E Capacity Development

Seed Funds for the UEA

PRSC3 policy matrix

PRSC2 policy matrix

PRSC1 policy matrix

Agenda of evaluations, studies, reviews

Regular service delivery surveys

PEAP revision

PEAP revision

• Streamlining of M&E responsibilities within the GoU • Strengthened role of civil society in monitoring

government performance • Improved results orientation of sector planning • More comprehensive scope of national budgeting process • Simplification of inspection and data collection • Increased emphasis on cost-effectiveness, value for money• Improved availability of information for decision-making

2003 2000 2001 2002

National M&E outputs contributed to:

Budget guideline focus on outcomes

Expanded schedule of PETS

UEA established

M&E Working Group established

Local gov’t information collection system

Strategy to consolidateinspection

Cabinet paper on coordination

Poverty M&E Strategy

M&E Progress Report

Seminar w/ Pov M&E Network

Hague seminar on PRSP M&E

CDF evaluation mission

High-level M&E meeting

Civil society M&E workshop

ECD conf Johannes-burg

AfrEA conf Nairobi

IPDET training

IPDET training

IPDET ttraining

Sector M&E report

M&E diagnosis report

First OED ECD mission

National M&E outcomes contributed to:

OED M&E inputs and activities:

19

Contribution of OED to national M&E output: Output: Identification of key M&E challenges and opportunities Indicator: M&E diagnosis report published in 2001 Annual revision of PRSC/PRSO policy matrix Sector M&E study published in 2002 Lessons learned:

• There is a need to build on substantive work already under way in the country—for example, the Poverty Monitoring Analysis Unit (of MOFPED), PEAP M&E work.

• OED analytical work has been critical to identification of M&E as a priority, cross-cutting public management issue.

• M&E benchmarks within the PRSC policy matrix, for which OED’s contribution played a catalytic role, have become the de facto agenda of M&E issues to be addressed.

• Substantive materials and reports lend credibility to arguments for change. However, senior decision-makers are often overburdened by documentation. In terms of the studies done as part of M&E work, it has often been oral presentations, rather than documents per se, that are most effective in influencing decision-makers. In addition to economizing on time, making a presentation also has a positive dynamic in public, open debate.

• “Bright ideas” from outside are often justifiably resisted. Instigating reforms is frequently dependent on persuading a critical mass of influential insiders who can carry an issue forward to a specialist audience.

• National M&E practices must be congruent with country decision-making habits and needs. However, although national ownership is paramount in success of reform efforts, responding to external pressures represents part of the reality faced by aid-dependent developing countries.

Output: Pool of national M&E skills expanded Indicator: Number of Ugandans completing IPDET Membership in the UEA Lessons learned:

• Twenty-one Ugandan officials have participated in the 2001, 2002, and 2003 IPDET programs. These include representatives of key GoU central and technical ministries and CSOs.

• OED funding support and training activitiesfor example, IPDET scholarships, support for participants to attend AfrEA conferences, the Johannesburg and Abidjan ECD seminars, and financial support for the UEAneed to be undertaken early and in a strategic manner, such as in terms of targeted participant selection.

20

• Beyond acquisition of skills, M&E training also had a policy impact. Participation of senior officials, such as the Deputy Permanent Secretary and Director of Budget (MOFPED), and the Assistant Commissioner for M&E (OPM), in IPDET and participation of MOFPED and OPM staff in regional ECD seminars have been critical to appreciation of M&E as an area of cross-cutting public management reform.

• Ugandans’ participation in OED’s regional or global events also had positive effects for networking between professionals from different countries.

• A large number of Ugandan civil servants and NGO professionals are eager to undertake M&E training, too.

• The development of M&E in Uganda has been hampered by an over-supply of disparate and parallel systems and practices of information collection and management. The multiplicity of donor M&E requirements and formats has contributed to undermining emergence of national capacities.

OED’s contribution to Uganda M&E outcomes: Outcome: Streamlining of responsibilities within the GoU Indicators: Poverty Monitoring and Evaluation Strategy formulated

Cabinet paper on M&E responsibilities prepared High-level M&E working group constituted M&E defined as cross-cutting reform area within the PRSC policy matrix Lessons learned:

• M&E coordination is associated with power, and resolving responsibilities requires political-level decisions, which are inevitably sensitive and even painful for some.

• Ownership of the M&E agenda has been helped by the initial demand for Bank assistance with M&E from within the GoU, rather than the Bank.

• There is strong demand for coordination responsibility from the OPM. MOFPED appears ready to cede functions that it has served to date.

• As M&E becomes recognized as important, individual central ministries have a tendency to press for their own whole-of-government version or vision (for example, the OPM and the Ministry of Local Government. There is a danger that this might be adding complexity rather than facilitating simplification of M&E.

• Some problems are most fruitfully brought to the fore by external actors who are not part of local networks of alliances. Although the availability of travel allowances can be a factor motivating excessive inspection, for instance, and this is immediately recognized by GoU officials in general, it appears that its recognition as a problem has needed highlighting by outside analysts. An external evaluation that documents or demonstrates an issue that may already be understoodfor example, PETS (which identified “leakage”) and inspection travel allowances (the three-sector review)has considerable credibility and catalyses focus and action to address the problem.

• In addition to key national champions to drive reforms within the country, it is important that there are also such champions in the Bank country team to ensure that M&E is integrated with other processes of Bank interaction with the country (for example, the PRSC, public expenditure reviews, the Bank’s Country Assistance

21

Strategy). The long-term nature of ECD will require a succession of Bank champions—any break in this succession would be a danger.

• Responsibilities continually evolve. For example, in Uganda it is not yet clear what role the newly established National Planning Authority will assume—any split of the planning and finance functions would be very unfortunate. It is therefore imperative that flexibility is maintained in terms of substantive focus and constituency of interaction. The M&E reform agenda will need to accommodate the mandate that will evolve for the National Planning Authority. This will also include their eventual involvement in the M&E coordination function.

Outcome: Strengthened role of civil society in monitoring government performance Indicator: Civil society participation in sector planning Public availability of M&E information Establishment of pilot community and client scorecard mechanism Lessons learned:

• It takes time for CSOs to realize what role they can most meaningfully play in monitoring public service delivery. In Uganda the National NGO Forum plays an important role in bringing civil society stakeholders together.

• The Bank has played a valuable role in showcasing to civil society successful examples from other countries of civil society (NGOs, think-tanks) assessing government performance (for example, client, citizen, or community scorecards akin to that administered by the Bangalore Public Affairs Centre).

• Although such showcasing has evidently been successful in building awareness and demand among civil society in Uganda, such mechanisms and instruments need to be tailored to local circumstances.

• Civil society is a ready supporter of M&E, but only the more capable and confident NGOs are prepared to actually get involved in critically monitoring government performance and service delivery (for example, the Uganda Debt Network’s piloting of community-based M&E).

Outcome: Improved results orientation of sector planning Indicator: Budget guidelines reformulated Outcomes indicators adopted as focus of ministerial planning Lessons learned:

• In the education sector indicators of learning outcomes have recently (2002–03) emerged as key performance measures, as evidenced by the SWAP joint review meeting and Budget Framework Paper.

• Improving clarity and results orientation of planning efforts is not only a matter of logical appreciation: It is as much about building comfort and consensus among sectors and other stakeholders, which invariably takes time.

• Ever since the first M&E diagnosis in 2001, a clearer focus on learning outcomes has been a continuous OED message in interaction with education sector professionals.

22

• Liaison by OED’s ECD consultant with the Bank’s education sector teamrather than direct contact with, for example, ministry officials—appears to have been most effective in achieving progress on this front.

• In swaying policy-level officials within a technical specialist culture, the Bank sector teams, who are long-standing insiders to the SWAP process, bring more credibility and weight than an outside M&E expert.

• Dealing with demand for M&E shows the issue to be inseparable from broader concerns with results orientation, public governance, and civil service reform.

• Affecting change in demand requires sustained advocacy and engagement with multiple sets of decision-making constituencies.

• Strong government commitment to a broad public sector reform agenda has proved to be a critical success factor for M&E capacity development.

Outcome: Simplification of inspection and data collection Indicator: Strategy for consolidation of inspection announced

Establishment of local government information collection system Lessons learned:

• Documenting (evaluating) a problem issue (a) legitimises and validates its existence as a problem and (b) focuses the attention of GoU and donor stakeholders on it.

• Through the three-sector M&E study, policy-makers now understand that current practices overload managers with inspection and poorly used data collection.

• There are increased risks of data redundancy with multiple, separate M&E systems. • Data redundancy is likely to undermine data quality. When it is known that much of

the data collected are not in any case used, there is little point to the effort needed to ascertain their timeliness and accuracy.

• New and harmonized practices and systems will bring about simplification only if old legacy practices and systems are abolished.

• Because there are incentives tied to existing M&E practices, such as travel allowances, there is resistance to abolishing redundant systems.

• Supply of M&E responds to incentives. Where rewards are given for undertaking M&E, as opposed to the actual performance revealed, activity will increase without necessarily effecting any improvement in the quality of decision-making. In Uganda there are several areas of practice where staff are rewarded for tidy paperwork.

• Systems development efforts must be anchored in a strategic, policy-level perspective of M&E use in the decision-making process.

• Poor data quality is a perennial “sleeper” and is not yet well addressed by GoU or the Bank.

23

Outcome: Increased emphasis on cost-effectiveness, value for money Indicator: Expanded schedule of expenditure tracking studies Use of NSDS data Lessons learned:

• Three expenditure tracking studies are being undertaken in 2003 (health, education, and water). This represents a considerable increase from one in each of the two preceding years.

• In retrospect it can be seen that the 2000 NSDS survey was conceived as “yet another” study, without a clear picture of how it would feed into decision-making processes. It is important to beware of supply-driven approaches and avoid the mindset that evaluation reports have inherent worth.

• Following a continuous focus within the PRSC, the 2003 NSDS survey, which is to be undertaken by UBOS, is being redesigned to align survey coverage with ministry responsibility. Such alignment is a precondition of the survey data being useful as a baseline for setting performance improvement targets, client service standards, or both.

• Decisions to undertake PETS have been made in the context of SWAP and the sector working group. Although this is still largely a result of demand from donor partners, an awareness is emerging among national stakeholders of the value of such studies.

Outcome: More comprehensive scope of national budget planning process Indicator: Increased share of donor funding included in the national budget and medium-term

expenditure framework (MTEF) Fewer stand-alone donor projects with separate M&E requirements Estimates of project assistance included in sector budget ceilings Lessons learned:

• Donors will substitute their own M&E requirements for those of national authorities only when they feel assured that the latter represent reasonable standards of accountability and fiduciary control.

• In Uganda the World Bank’s shift toward PRSCs/PRSOs as the modality of engagement has been instrumental in other donors gaining comfort with, and participating in the provision of, programmatic and budget support.

• The sector working group mechanism, as a forum for joint stakeholder planning and consultation, has been critical to donor buy-in to budget support.

• Although the GoU has included estimates for project assistance in sector budget ceilings, not all donor projects are realistically costed.

24

AN

NE

X D

. M&

E in

Uga

nda’

s PR

SO3

Dra

ft P

olic

y M

atri

x (a

s of A

pril

2003

)(i)

Ben

chm

arks

for

M&

E

AC

TIO

NS

AN

D O

UT

PUT

TA

RG

ET

S

PRSO

3 PR

SO4

PRSO

5

IS

SUE

Apr

il 1,

200

3 A

pril

1, 2

004

Apr

il 1,

200

5 O

UTC

OM

ES

Ove

rall

PRSO

obj

ectiv

e: Im

prov

ed b

asic

serv

ice

deliv

ery

Ove

rall

outc

omes

(ver

ifica

tion)

: 1.

Red

uctio

n in

hea

dcou

nt in

dex

of p

over

ty (a

nnua

l hou

seho

ld su

rvey

s).

2. In

crea

sed

satis

fact

ion

with

pub

lic se

rvic

es (n

atio

nal s

ervi

ce d

eliv

ery

surv

eys a

nd p

erio

dic

parti

cipa

tory

pov

erty

as

sess

men

ts)

PEA

P/PR

SP P

ILLA

R I:

FR

AM

EWO

RK

FO

R E

CO

NO

MIC

GR

OW

TH A

ND

TR

AN

SFO

RM

ATI

ON

PR

SO O

bjec

tive

A: E

ffic

ient

and

equ

itabl

e us

e of

pub

lic re

sour

ces

Ove

rall

outc

ome

(ver

ifica

tion)

: 1.

Mor

e eq

uita

ble

acce

ss to

pub

lic se

rvic

es (i

ncid

ence

ana

lysi

s usi

ng h

ouse

hold

surv

eys a

nd p

ublic

exp

endi

ture

da

ta).

Res

ults

-ori

enta

tion

and

Mon

itori

ng a

nd E

valu

atio

n (M

&E

) Se

ctor

pla

ns a

nd

expe

nditu

re p

rogr

ams a

re

inpu

t-orie

nted

.

Roa

ds a

nd a

gric

ultu

re h

ave

inco

rpor

ated

out

put a

nd o

utco

me

mea

sure

s and

targ

ets i

n th

e 20

03-0

4 to

20

05-0

6 B

FPs.

Bas

ed o

n se

ctor

wor

king

gro

up d

ecis

ions

, ad

ditio

nal s

ecto

rs in

corp

orat

e ou

tput

and

ou

tcom

e m

easu

res a

nd ta

rget

s in

the

2004

-05

to 2

006-

07 B

FPs

Bas

ed o

n se

ctor

wor

king

gro

up

deci

sion

s, ad

ditio

nal s

ecto

rs

inco

rpor

ate

outp

ut a

nd o

utco

me

mea

sure

s and

targ

ets i

n th

e 20

05–0

6 to

200

7–08

BFP

s.

Sect

or p

lans

and

exp

endi

ture

pr

ogra

ms a

re o

utpu

t/out

com

e or

ient

ed.

Lack

of c

lear

div

isio

n of

la

bor i

n M

&E.

M

OFP

ED a

nd O

PM h

ave

prep

ared

a

Cab

inet

Info

rmat

ion

Pape

r on

stre

ngth

enin

g co

ordi

natio

n in

pub

lic

sect

or m

anag

emen

t.

OPM

, MO

FPED

, MO

PS, M

OLG

, and

ot

her m

inis

tries

est

ablis

h an

inte

grat

ed

M&

E sy

stem

to su

ppor

t the

coo

rdin

atio

n ar

rang

emen

ts.

C

larit

y of

M&

E ro

les a

nd

func

tions

.

OPM

, in

cons

ulta

tion

with

MO

FPED

, M

OPS

, MO

LG, a

nd li

ne m

inis

tries

, ha

s agr

eed

on a

n ap

proa

ch to

co

nsol

idat

e G

oU in

spec

tions

.

OPM

, MO

FPED

, MO

PS, M

OLG

, and

lin

e m

inis

tries

star

t con

solid

atin

g in

spec

tions

.

Ef

fect

iven

ess o

f ser

vice

de

liver

y un

derm

ined

by

mul

tiple

M&

E ar

rang

emen

ts.

MO

PS, M

OFP

ED, M

OLG

, OPM

, lin

e m

inis

tries

, and

UB

OS

have

agr

eed

on

com

mon

resu

lts m

anag

emen

t con

cept

s an

d de

finiti

ons.

OPM

, MO

FPED

, MO

PS, M

OLG

, and

lin

e m

inis

tries

agr

ee o

n an

act

ion

plan

for

a co

mm

on d

ata

colle

ctio

n sy

stem

.

OPM

, MO

FPED

, MO

PS, M

OLG

, an

d lin

e m

inis

tries

iden

tify

a re

duce

d se

t of i

ndic

ator

s for

loca

l go

vern

men

t dat

a co

llect

ion.

Coo

rdin

ated

and

sim

plifi

ed

data

col

lect

ion

and

insp

ectio

n ac

tivity

.

Nee

d fo

r mor

e sy

stem

atic

ap

proa

ch to

eva

luat

ions

, va

lue

for m

oney

and

tra

ckin

g st

udie

s.

MO

FPED

, in

cons

ulta

tion

with

OPM

an

d ot

her m

inis

tries

, has

com

pile

d an

ag

enda

of o

ngoi

ng a

nd p

lann

ed

eval

uatio

ns, v

alue

for m

oney

and

tra

ckin

g st

udie

s.

MO

FPED

toge

ther

with

OPM

and

oth

er

min

istri

es, d

evel

ops a

pla

n fo

r fur

ther

ev

alua

tions

and

stud

ies,

and

impr

oves

m

etho

dolo

gies

use

d.

R

evie

w m

echa

nism

s sup

port

effe

ctiv

enes

s of p

ublic

serv

ice

deliv

ery.

Nee

d fo

r inc

reas

ed u

se o

f be

nefic

iary

ass

essm

ents

to

prom

ote

publ

ic

acco

unta

bilit

y.

MO

PS, M

OFP

ED, M

OLG

, OPM

, lin

e m

inis

tries

, and

UB

OS

have

toge

ther

re

desi

gned

the

natio

nal s

ervi

ce

deliv

ery

surv

ey

MO

P, to

geth

er w

ith o

ther

min

istri

es,

esta

blis

hes s

ervi

ce d

eliv

ery

stan

dard

s ba

sed

on n

atio

nal s

ervi

ce d

eliv

ery

resu

lts.

B

enef

icia

ryas

sess

men

tsin

tegr

ated

in p

lann

ing

and

perf

orm

ance

mon

itorin

g.

25

AN

NE

X D

. M&

E in

Uga

nda’

s PR

SO3

Dra

ft P

olic

y M

atri

x (a

s of A

pril

2003

)(ii

) Oth

er B

ench

mar

ks R

elev

ant t

o M

&E

A

CT

ION

S A

ND

OU

TPU

T T

AR

GE

TS

PRSO

3 PR

SO4

PRSO

5

IS

SUE

Apr

il 1,

200

3 A

pril

1, 2

004

Apr

il 1,

200

5 O

UTC

OM

ES

Ove

rall

PRSO

obj

ectiv

e: Im

prov

ed b

asic

serv

ice

deliv

ery

Ove

rall

outc

omes

(ver

ifica

tion)

: 1.

Red

uctio

n in

hea

dcou

nt in

dex

of p

over

ty (a

nnua

l hou

seho

ld su

rvey

s).

2. In

crea

sed

satis

fact

ion

with

pub

lic se

rvic

es (n

atio

nal s

ervi

ce d

eliv

ery

surv

eys a

nd p

erio

dic

parti

cipa

tory

pov

erty

as

sess

men

ts).

PEA

P/PR

SP P

ILLA

R I:

FR

AM

EWO

RK

FO

R E

CO

NO

MIC

GR

OW

TH A

ND

TR

AN

SFO

RM

ATI

ON

PR

SO o

bjec

tive

A: E

ffic

ient

and

equ

itabl

e us

e of

pub

lic re

sour

ces

Ove

rall

outc

ome

(ver

ifica

tion)

: 1.

M

ore

equi

tabl

e ac

cess

to p

ublic

serv

ices

(inc

iden

ce a

naly

sis u

sing

hou

seho

ld su

rvey

s and

pub

lic

expe

nditu

re d

ata)

. A

lloca

tions

and

act

ual e

xpen

ditu

re:

Res

ourc

es n

eed

to fl

ow in

a

timel

y an

d pr

edic

tabl

e m

anne

r to

sect

ors a

nd

dist

ricts

.

MO

FPED

, tog

ethe

r with

rele

vant

line

m

inis

tries

, has

con

duct

ed re

view

s on

prob

lem

s rel

ated

to re

sour

ce fl

ows a

nd

take

n re

med

ial a

ctio

n.

MO

FPED

, tog

ethe

r with

rele

vant

line

m

inis

tries

, con

duct

s rev

iew

s on

prob

lem

s rel

ated

to re

sour

ce fl

ows a

nd

take

s rem

edia

l act

ion.

MO

FPED

, tog

ethe

r with

rele

vant

lin

e m

inis

tries

, con

duct

s rev

iew

s on

prob

lem

s rel

ated

to re

sour

ce fl

ows

and

take

s rem

edia

l act

ion.

Tim

ely

and

pred

icta

ble

reso

urce

flow

s to

fron

tline

se

rvic

e de

liver

y.

M

inis

try o

f Hea

lth (M

OH

), M

inis

try o

f Ed

ucat

ion

and

Spor

ts (M

OES

), an

d M

OW

LE (D

irect

orat

e of

Wat

er

Dev

elop

men

tD

WD

) hav

e ca

rrie

d ou

t tra

ckin

g st

udie

s and

take

follo

w-u

p ac

tions

.

MO

H, M

OES

, MO

WLE

/DW

D, a

nd

the

just

ice,

law

and

ord

er (J

LO) s

ecto

r ca

rry

out t

rack

ing

stud

ies a

nd ta

ke

follo

w-u

p ac

tions

.

MO

H, M

OES

, MO

WLE

/DW

D, a

nd

JLO

car

ry o

ut tr

acki

ng st

udie

s and

ta

ke fo

llow

-up

actio

ns.

Ann

ual o

uttu

rns h

ave

devi

ated

from

app

rove

d bu

dget

s.

MO

FPED

has

pub

lishe

d se

mi-a

nnua

l bu

dget

per

form

ance

repo

rts a

nd m

ade

quar

terly

repo

rts a

vaila

ble

on re

ques

t.

MO

FPED

pub

lishe

s sem

i-ann

ual

budg

et p

erfo

rman

ce re

ports

and

mak

es

quar

terly

repo

rts a

vaila

ble

on re

ques

t.

MO

FPED

pub

lishe

s sem

i-ann

ual

budg

et p

erfo

rman

ce re

ports

and

m

akes

qua

rterly

repo

rts a

vaila

ble

on

requ

est.

Bud

get e

xecu

tion

is c

lose

r to

the

appr

oved

app

ropr

iatio

ns.

Inte

rgov

ernm

enta

l tra

nsfe

rs:

Infle

xibi

lity

in th

e us

e of

co

nditi

onal

gra

nts w

eake

ns

loca

l ow

ners

hip

and

impo

ses

burd

enso

me

repo

rting

re

quire

men

ts.

MO

FPED

, MO

LG, a

nd th

e Lo

cal

Gov

ernm

ent F

inan

ce C

omm

issi

on

(LG

FC) h

ave

prep

ared

ope

ratio

nal

man

uals

and

gui

delin

es fo

r im

plem

entin

g th

e fis

cal

MO

FPED

and

MO

LG im

plem

ent t

he

stra

tegy

in p

ilot d

istri

cts i

n 20

03–0

4.

MO

FPED

and

MO

LG im

plem

ent t

he

stra

tegy

in a

ll di

stric

ts in

200

4–05

. R

educ

ed a

dmin

istra

tive

cost

s an

d in

crea

sed

loca

l ow

ners

hip

of e

xpen

ditu

re p

rogr

ams.

Publ

ic se

rvic

e m

anag

emen

t:

Publ

ic se

rvic

e re

form

has

no

t yet

yie

lded

maj

or

impr

ovem

ents

in se

rvic

e de

liver

y.

MO

PS h

as p

rese

nted

a st

rate

gy fo

r the

ne

w p

hase

of t

he p

ublic

serv

ice

refo

rm

prog

ram

to a

bine

t for

app

rova

l.

MO

PS im

plem

ents

the

new

pha

se o

f th

e pu

blic

serv

ice

refo

rm p

rogr

am.

MO

PS re

view

s pro

gres

s in

impl

emen

ting

the

publ

ic se

rvic

e re

form

pro

gram

.

Impr

oved

pub

lic se

rvic

e de

liver

y.

26

AN

NE

X D

. M&

E in

Uga

nda’

s PR

SO3

Dra

ft P

olic

y M

atri

x (a

s of A

pril

2003

)(ii

) Oth

er B

ench

mar

ks R

elev

ant t

o M

&E

A

CT

ION

S A

ND

OU

TPU

T T

AR

GE

TS

PRSO

3 PR

SO4

PRSO

5

IS

SUE

Apr

il 1,

200

3 A

pril

1, 2

004

Apr

il 1,

200

5 O

UTC

OM

ES

MO

PS, i

n co

llabo

ratio

n w

ith li

ne

min

istri

es, h

as su

bmitt

ed p

relim

inar

y fin

ding

s on

cost

-eff

ectiv

enes

s and

ef

ficie

ncy

of e

mpl

oym

ent a

nd st

aff

utili

zatio

n in

soci

al se

ctor

s.

MO

PS o

btai

ns C

abin

et a

ppro

val o

f po

licy

for r

egul

atin

g an

d co

ntro

lling

th

e si

ze o

f pub

lic a

dmin

istra

tion

and

soci

al se

ctor

est

ablis

hmen

ts.

Rap

id e

xpan

sion

in p

ublic

ad

min

istra

tion

and

serv

ice

thre

aten

ing

to je

opar

dize

th

e ac

hiev

emen

ts o

f the

pu

blic

serv

ice

refo

rm.

MO

PS, i

n co

nsul

tatio

n w

ith

MO

FPED

, has

com

plet

ed a

n as

sess

men

t of t

he im

plic

atio

ns o

f ex

pand

ed p

ublic

adm

inis

tratio

n on

the

wag

e bi

ll.

MO

PS re

gula

tes a

nd c

ontro

ls g

row

th

in p

ublic

adm

inis

tratio

n an

d so

cial

se

ctor

est

ablis

hmen

ts.

The

grow

th o

f pub

lic

adm

inis

tratio

n an

d se

rvic

e is

co

ntai

ned.

The

pres

ent m

anua

l sys

tem

is

inad

equa

te fo

r acc

urat

e re

cord

ing

and

repo

rting

of

finan

cial

tran

sact

ions

.

MO

FPED

has

app

rove

d th

e de

sign

of a

n in

tegr

ated

fina

ncia

l man

agem

ent s

yste

m

(IFM

IS),

begu

n co

ntra

ctin

g ex

tern

al

supp

lier(

s), a

nd d

eplo

yed

addi

tiona

l ac

coun

ting

staf

f (gr

adua

tes)

to a

ssis

t in

impl

emen

tatio

n.

Five

min

istri

es (i

nclu

ding

MO

FPED

, M

OES

, MO

H, a

nd M

OW

LE) a

nd th

ree

loca

l gov

ernm

ents

pilo

t the

acc

ount

ing

mod

ule

of th

e IF

MIS

.

Furth

er m

inis

tries

and

loca

l go

vern

men

ts p

ilot t

he a

ccou

ntin

g m

odul

e of

the

IFM

IS.

Tim

ely

and

relia

ble

acco

unts

pr

oduc

ed b

y se

ctor

min

istri

es

in c

ompl

ianc

e w

ith g

ener

ally

ac

cept

ed a

ccou

ntin

g pr

inci

ples

.

Tra

nspa

renc

y:

Lim

ited

acce

ss to

pub

lic

info

rmat

ion

to e

mpo

wer

ci

tizen

s to

dem

and

grea

ter

acco

unta

bilit

y.

MO

PS h

as c

ondu

cted

con

sulta

tions

on

acce

ss to

gov

ernm

ent i

nfor

mat

ion

legi

slat

ion.

MO

PS ta

bles

the

prin

cipl

es o

f acc

ess

to g

over

nmen

t inf

orm

atio

n to

the

Cab

inet

.

MO

PS ta

bles

the

Acc

ess t

o G

over

nmen

t Inf

orm

atio

n B

ill to

Pa

rliam

ent.

Impr

oved

acc

ess t

o go

vern

men

t in

form

atio

n to

redu

ce

oppo

rtuni

ties f

or c

orru

ptio

n.

Det

ectio

n, in

vest

igat

ion,

and

pro

secu

tion

of c

orru

ptio

n ca

ses:

C

orru

pt a

cts a

re n

ot e

xpos

ed

and

sanc

tione

d ef

fect

ivel

y.

The

Dire

ctor

ate

of E

thic

s and

Inte

grity

(D

EI) h

as p

rese

nted

prin

cipl

es o

f the

Pr

even

tion

of C

orru

ptio

n B

ill to

the

Cab

inet

.

The

DEI

tabl

es th

e Pr

even

tion

of

Cor

rupt

ion

Bill

in P

arlia

men

t.

Mor

e ef

fect

ive

follo

w-u

p of

co

rrup

tion

case

s.

The

Insp

ecto

r Gen

eral

of G

over

nmen

t an

d D

EI e

nsur

e th

at m

onito

ring

of

corr

uptio

n be

com

es p

art o

f M&

E.

R

egul

ar m

onito

ring

of

corr

uptio

n.

Civ

il so

ciet

y pa

rtic

ipat

ion:

Pa

rtici

pato

ry a

ppro

ache

s in

serv

ice

deliv

ery

are

unco

ordi

nate

d.

Loca

l gov

ernm

ents

hav

e im

plem

ente

d th

e ha

rmon

ized

par

ticip

ator

y pl

anni

ng

guid

e (H

PPG

) for

low

er lo

cal c

ounc

ils.

MO

LG a

nd lo

cal c

ounc

ils m

onito

r the

pe

rfor

man

ce o

f the

HPP

G.

MO

LG a

nd lo

cal c

ounc

ils re

view

the

impl

emen

tatio

n of

the

HPP

G.

Serv

ice

deliv

ery

resp

ondi

ng to

us

er p

refe

renc

es.

27

ANNEX E. Select Findings from the Study of M&E in Uganda’s Health, Education, and Water Sectors6 Terminology of M&E Sector ministries have been tasked with identifying indicators and setting goals and targets, thus terminology and approach vary among the sectors. Some use the terms outputs and outcomes interchangeably to denote “what comes after,” whereas others are clearer about the difference. There are different notions of what constitutes performance. Presentation and timeframe for results information are not uniform. Sector results indicators For each of the three Ugandan service sectors, there are several different sets of goal or priority indicators highlighted in the documents that capture sector strategy, for example, the PMES, PRSC, PEAP, PRSP, sector policy documents, and measures used in BFPs (see table E.1).

Table E.1. Different sets of sector goal and priority indicators MDGs PMES

priority indicators

PRSC PEAP 2001–03

PRSP Sector policy plan/strategy

BFP “physical performance”

No. Type/focus

No. Type/focus

No. Type/ focus

No. Type/focus

No. Type/ focus

No. Type/focus

No. Type/ focus

Health 15 Outcome 8 Outcome/access, process

2 Outcome/access

20 Mixed 2 Process

Education 5 Outcome 5 Mainly process + access

2 Outcome/access

13 Mixed 3 Process

Water 2 Access 2 Access 2 Outcome/access

7 Mixed 3 Process

20+14 15

Mainly process + access Mainly process + access

5 5 9

Access + process Process +1 access Process + 1 access

Ministries, districts and PAF reporting Liaison between the sector ministries and districts focuses on PAF resources and occurs through lines of communication established between ministries and their respective technical representatives in the district administration, with information only nominally flowing through the chief accounting officer and the office of the district economist/planner. This practice is mirrored in communications between the district and the sub-county. In the health sector there are fairly extensive contacts directly between the ministry and the health sub-district team, that is, by-passing both the district and sub-district management structures. The risk is that districts maintain no view of their own needs or of cross-sectoral issues. Local governments (LGs) have little decision-making power, given that the bulk of their allocations are protected under PAF and provided as conditional grants for which areas of expenditure are tightly prescribed. Although the conditional grant mechanism is helpful in enforcing accountability, it comes at the expense of local autonomy in tailoring services to local needs. In practice it has the effect of reducing the LGs to the role of an implementing agency for the respective central ministries. 6 Hauge et al (2002).

28

District data collection Districts have the main responsibility for data collection for ministerial MISs. Data compiled at the district level amount to a substantial volume. The three sector MISs entail collection of data on nearly 1,000 different indicators, with almost 300,000 data entries per year )Tsble E.2). Health and education sector MISs cover sectoral service delivery and some records of expenditure. Performance indicators focus mainly on spending, activities, and state of physical facilities. Some of the data provided are of a poor quality, for example, with “ghost schools” and “ghost teachers” reported in the education sector. Outcome measures (for example, health status and learning outcomes) and client satisfaction perspectives are largely missing. As a result the MISs yield little information on cost-effectiveness of the service delivery system.

Table E.2. Data Workload for Districts in Compiling Sectoral Service Delivery Indicators District National Sector No. of

indicators collected

Avg. no. of units per district

frequency Avg. annual data entries per district

Total no. of service delivery units

Total no. of data records captured

Primary education 737 211 Annual 155,507 11,808 8,708,392 Health 229 29 Monthly 83,891 1,738 4,697,896 Water 33 440 Quarterly 58,080 24,640 3,252,480

Central and ministerial inspection If inspection and support visits are undertaken in accordance with the schedule envisaged by the health sector national supervision guidelines, a simple estimate is that the combined cumulative time spent on such missions would be in the region of 150,000 person-days per year, or the equivalent of 700 full-time posts. This estimate is for time spent by supervisors. If, additionally, the total amount of time spent on inspection is similar for supervisees, the combined workload involved would be equivalent to about 1,400 full-time posts, largely involving qualified medical personnel. Many of those interviewed pointed out that their time would be better spent on actually providing health care services. Within all sectors the prevalent view of M&E is one where the centre makes physical visits to the district or facility, with an objective of verifying content of district reports and making technical assessment of facilities. In all three sectors a number of the officials interviewed stated that they view M&E predominantly as an accountability and compliance function—as being “checked” by others rather than being useful by providing support to ongoing management. Staff at the facility and LG levels who provide data for sector MISs and who attend to inspectors and supervisor reported that they supply information because they are “told to” or because “that’s what the rules are.” They are unsure of how the information they supply is used. Little of it is used for any decision-making at the level of those involved in data collection. From the perspective of district and facility managers, the problems with current inspection and supervision activity include: (a) involvement of multiple inspection agencies, (b) uncoordinated inspection visits, (c) attending to inspectors that diverts time from productive activity, (d) poor feedback from inspection visits, and (e) the prime motivating factor behind inspection visits is collection of allowances by inspectors (see below).

29

Motivation for M&E For those who are suppliers of M&E information, the primary concern is to secure release of funds. LGs are penalized for late or inaccurate submission of PAF quarterly reports. Whereas timely submission of a report reflecting poor performance may secure release of funds, late submission of a report truthfully describing good performance may be penalized through delay in release of funds. Under the local government development program rewards are available to the LGs that have tidy books. Where the provision of M&E information has no tie to resource release, compliance is poor. For instance, in the system of project reporting that has been established by OPM, only 40 percent of reports due from subject ministries and 16 percent of all ministry reports were actually submitted in 2001. Inspectors from central accountability institutions, line ministries, districts, and other agencies collect substantial allowances for undertaking inspections. With typical daily travel allowances of Ush50,000 and monthly salaries in the region of USh300,000–500,000, field visits represent an opportunity for some staff to multiply their income. Within the MOWLE, for example, the combined estimated 2001–02 recurrent cost of domestic travel and non-wage allowances is USh883 million, equivalent to 44.8 percent of wage costs. These expenditures are not exclusively for M&E activities, but they illustrate the availability and potential attraction of allowances as a motivational force for inspection activity. Views of beneficiaries The end users or intended beneficiaries of sector services are poorly represented in health, education, and water sector M&E activity. There is little evidence of end user accountability mechanisms, such as water user groups or parent-teacher associations, being systematically consulted to ascertain that value for money is achieved in catering to client needs. Yet these groups have a more personal and immediate interest in the quality and accessibility of such services. They can regularly observe facilities such as school classrooms and clinics at close quarters, and they can undertake basic inspection tasks without needing to collect travel and meal allowances. Conversely, communities consider school and health management committees as vehicles of facility managers’ interests, sometimes to the detriment of clients or users; they are not empowered to question their leaders or service delivery managers.

30

References Booth, D., and H. Lucas. 2001. “Desk Study of Good Practice in the Development of PRSP

Indicators and Monitoring Systems: Interim and Final Reports.” Overseas Development Institute, London.

Brock, K., R. McGee, and R. Ssewakiryanga. 2002. “Poverty Knowledge and Policy Processes: A Case Study of Ugandan National Poverty Reduction Policy. Institute of Development Studies Research Report 52. Sussex.

Directorate of Ethics and Integrity. 2001. “Strategy and Plan of Action to Fight Corruption and Build Ethics and Integrity in Public Life.” Kampala.

Gariyo, Z. 2000. “Citizen Involvement in the Budgetary Process in Uganda.” Uganda Debt Network, Glasgow. Available at: http://www.worldbank.org/participation/ugandabudget.htm.

Hauge, A. 2001. “Strengthening Capacity for Monitoring and Evaluation in Uganda: A Results Based Management Perspective.” ECD Working Paper 8. World Bank, OED, Washington, D.C.

Hauge, A., W. Kisembo, M. Makumbi, and O. Omongin. 2002. “Effectiveness of Uganda’s PEAP: The Role of M&E in the Health, Education and Water Sectors.” MOFPED, Kampala.

Mackay, K. 1999. Evaluation Capacity Development: A Diagnostic Guide and Action Framework. ECD Working Paper No. 6. World Bank, OED, Washington, D.C.

———. 2002. Annual Report on Evaluation Capacity. World Bank, OED, Washington, D.C. MOPS (Ministry of Public Service). 2002. A Handbook on Results-Oriented Management: A

Guide for Public Service Managers. Kampala. MOFPED (Ministry of Finance, Planning and Economic Development). 2000. “Improving

Budget Transparency in Uganda: Informing Stakeholders and Including Them in the Budget Process.” Phase I Report. Kampala.

———. 2001. “PAF: General Guideline for the Planning and Operation of Conditional Grants 2001–2002.” Kampala.

MOFPEDPoverty Monitoring Network. 2002. “Poverty Monitoring and Evaluation Strategy.” Kampala.

MOFPEDPoverty Monitoring and Analysis Unit. 2003. Uganda Participatory Poverty Assessment 2. Kampala.

Reinikka, R., and J. Svensson 2002. “Assessing Frontline Service Delivery.” World Bank, Development Research Group, Washington, D.C.

31

32

Other Papers in This Series #1: Keith Mackay, Lessons from National Experience. #2: Stephen Brushett, Zimbabwe: Issues and Opportunities. #3: Alain Barberie, Indonesia’s National Evaluation System. #4: Keith Mackay, The Development of Australia’s Evaluation System. #5: R. Pablo Guerrero O., Comparative Insights from Colombia, China and Indonesia. #6: Keith Mackay, Evaluation Capacity Development: A Diagnostic Guide and Action

Framework. #7: Mark Schacter, Sub-Saharan Africa: Lessons from Experience in Supporting Sound

Governance. #8: Arild Hauge, Strengthening Capacity for Monitoring and Evaluation in Uganda: A

Results Based Management Perspective. #9: Marie-Hélène Adrien, Guide to Conducting Reviews of Organizations Supplying

M&E Training. Other Recommended Reading OED, 2002 Annual Report on Evaluation Capacity Development (AREC). OED, Monitoring and Evaluation: Some Tools, Methods and Approaches, 2002. Development Bank of Southern Africa, African Development Bank and The World Bank, Developing African Capacity for Monitoring and Evaluation, 2000. K. Mackay and S. Gariba (eds.), The Role of Civil Society in Assessing Public Sector Performance in Ghana, OED, 2000. Other relevant publications can be downloaded from OED’s ECD Website: >http://www.worldbank.org/oed/ecd/>