Annual Review - Summary Sheet
27
1 Annual Review - Summary Sheet This Summary Sheet captures the headlines on programme performance, agreed actions and learning over the course of the review period. It should be attached to all subsequent reviews to build a complete picture of actions and learning throughout the life of the programme. Title: Maternal, Newborn and Child Health 2 Programme Value: 85,910,949 Review Date: 18 May – 30 June Programme Code: 202992 Start Date: June 1, 2014 End Date: May 31, 2019 Summary of Programme Performance Year 1 st AR 18/7/13 to 31/7/14 2 nd AR 1/6/14 to 31/3/15 3 rd AR 1/4/15 to 31/3/16 Programme Score A A A Risk Rating Low Low Moderate Summary of progress and lessons learnt since last review MNCH2 has made good progress during its second year, attaining almost all outcome and output level targets. The programme has supported and further enhanced initiatives established by predecessors (eg emergency transport scheme, facility health committees) and has introduced new approaches and tools to further strengthen state and LGA capacity to plan, budget and manage primary health care and ensure the provision of quality reproductive, maternal and child health services (eg Organisation Capacity Assessment Tool, Quality of Care Assessment Tool). The programme has stepped-up engagement with the Federal Ministry of Health (FMoH) and National Primary Health Care Development Agency (NPHCDA) to ensure that that lessons learned from MNCH2 are institutionalized and best practices replicated. This has included technical support, collaboration and participation in meetings to share, develop and review manuals, tools, policies and guidelines and the evalution of interventions. Most critically, over the past year MNCH2 has engaged more actively with health sector policy formulation, implementation and monitoring at Federal level and has supported States to implement Federal policies and initiatives (eg Saving One Million Lives Payment by Results, contributions to reviews on the national immunisation programme, among others). MNCH2 is recognized by the FMoH as a strong partner, with the potential to contribute significantly to the govermnent’s vision to ensure 10,000 functional primary health care centres by 2018, as a critical platform to attain universal health coverage. At State level, MNCH2 has aligned well with government and community structures. Project interventions and workplans are developed in consultation with state government (examples include the selection of facilities for MNCH2 support, development of procurement plans for equipment, drugs and consumables, the development and roll out of training packages), and interventions are, as far as possible, conducted with government workers (eg health worker training, supportive supervision, facility assessments). As the project approaches its mid-term, priority should now be given to developing a sustainability plan and exit strategy to ensure that successful interventions are fully embedded within government systems and to ensure that programme successes are sustained beyond the lifetime of MNCH2. Progress against recommendations of last annual review The second annual review of MNCH2 was completed in August 2015. Specific recommendations of the review and progress made against those recommendations are set out below.
Transcript of Annual Review - Summary Sheet
1
Annual Review - Summary Sheet This Summary Sheet captures the headlines on programme performance, agreed actions and learning over the course of the review period. It should be attached to all subsequent reviews to build a complete picture of actions and learning throughout the life of the programme.
Title: Maternal, Newborn and Child Health 2
Programme Value: 85,910,949 Review Date: 18 May – 30 June
Programme Code: 202992
Start Date: June 1, 2014
End Date: May 31, 2019
Summary of Programme Performance
Year 1st AR 18/7/13 to 31/7/14
2nd AR 1/6/14 to 31/3/15
3rd AR 1/4/15 to 31/3/16
Programme Score A A A
Risk Rating Low Low Moderate
Summary of progress and lessons learnt since last review MNCH2 has made good progress during its second year, attaining almost all outcome and output level targets. The programme has supported and further enhanced initiatives established by predecessors (eg emergency transport scheme, facility health committees) and has introduced new approaches and tools to further strengthen state and LGA capacity to plan, budget and manage primary health care and ensure the provision of quality reproductive, maternal and child health services (eg Organisation Capacity Assessment Tool, Quality of Care Assessment Tool). The programme has stepped-up engagement with the Federal Ministry of Health (FMoH) and National Primary Health Care Development Agency (NPHCDA) to ensure that that lessons learned from MNCH2 are institutionalized and best practices replicated. This has included technical support, collaboration and participation in meetings to share, develop and review manuals, tools, policies and guidelines and the evalution of interventions. Most critically, over the past year MNCH2 has engaged more actively with health sector policy formulation, implementation and monitoring at Federal level and has supported States to implement Federal policies and initiatives (eg Saving One Million Lives Payment by Results, contributions to reviews on the national immunisation programme, among others). MNCH2 is recognized by the FMoH as a strong partner, with the potential to contribute significantly to the govermnent’s vision to ensure 10,000 functional primary health care centres by 2018, as a critical platform to attain universal health coverage. At State level, MNCH2 has aligned well with government and community structures. Project interventions and workplans are developed in consultation with state government (examples include the selection of facilities for MNCH2 support, development of procurement plans for equipment, drugs and consumables, the development and roll out of training packages), and interventions are, as far as possible, conducted with government workers (eg health worker training, supportive supervision, facility assessments). As the project approaches its mid-term, priority should now be given to developing a sustainability plan and exit strategy to ensure that successful interventions are fully embedded within government systems and to ensure that programme successes are sustained beyond the lifetime of MNCH2. Progress against recommendations of last annual review The second annual review of MNCH2 was completed in August 2015. Specific recommendations of the review and progress made against those recommendations are set out below.
2
Table 1: Summary of progress against recommendations of Second Annual Review
RECOMMENDATION SUMMARY OF PROGRESS
1 DFID and MNCH2 should agree on targets that are not only ambitious but based on evidence, including accurate baselines, as far as possible. The agreed milestones should not only reflect the prerequisite effort required to sustain DFID’s long-term investment but deliver a step change in the MNCH indices in the respective states.
Completed. Baseline figures were revised based on UN population projections and DHS 2013 coverage rates. For each outcome indicator a formula was devised to review and re-set outcome targets annually to reflect a “rolling” baseline, whereby the new baseline will be the previous year’s achievement minus the MNCH2 contribution, scaled to match over/under performance against the prior baseline target. Formulae are set out in ‘Project Indicator Reference Sheets’ for each logframe indicator.
2 MNCH2 should develop a plan with SMOH and facilities to enhance Primary Health Care Under One Roof (PHCUOR) and agree indicators to measure success.
Completed. MNCH2, jointly with State Goverments and partners, developed a tool to measure the status of State Primary Health Care Development Agencies (SPHCDAs) in meeting PHCUOR, using indicators from the national guideline on implementation of PHCUOR. Each State was supported to assess the status of its agency, obtained a baseline and developed a Capacity Development Plan. MNCH2 has supported States in the implementation of their CDPs and a plan has been agreed to repeat the assessments annually to track progress.
3 MNCH2 should support facilities and LGAs to use the findings of Client Satisfaction Exit Interviews to tangibly inform programme planning, and inform topical research in the programme’s OR plan.
Completed. Client Satisfaction Exit Interviews were completed and shared with facilities and LGAs. Questions from client exit interviews now included in Community Scorecard – see comments below under Output 1, Indicator 1.2 for further detail.
4 DFID should ensure that lessons learned from the approaches taken by MNCH2 to strengthen citizen engagement (including scorecards, client surveys, facility health committees and OCAT) are shared widely within DFID and inform DFID Nigeria strategic planning and programming.
Ongoing. Lessons learned shared regularly between programmes at quarterly national health programme managers meetings and at state health coordination meetings. Meetings between MNCH2 and the new Partnership to Engage Reform and Learn (PERL) programme have also taken place to share learning and inform the design and activities of PERL.
5 MNCH2 should include a clear dissemination process in its OR Plan for sharing findings with State and Federal Ministries of Health and community stakeholders.
Completed. A dissemination strategy has been devised and implementation initiated. Strategy sets out key stakholders, and means of engagement for the review, dissemination and discussion on research findings. In addition to MNCH2 programme staff, stakeholders include health care managers, health service providers, and policy makers at LGA, state and national levels. Research products are tailored for the specific audience (eg full reports, technical and policy briefings, fact sheets as relevant). Research is submitted for publication in open-access peer reviewed journals.
6 MNCH2 should thoroughly review and establish a consistent understanding/perspective of ‘value for money’, which needs to be contextually based on the different states and LGAs. This includes developing a set of core economic questions, a rigorous sampling methodology and a robust approach to data collection to strengthen key economic measures for monitoring, programmatic decision-making, and eventually making regional and international comparisions.
Completed. A revised VfM strategy has been developed including a framework for assessing VfM throughout the programme cycle. A set of economic questions were identified. These will be measured through eleven indicators, one off analyses, case studies and qualitative evidence (see Section D below)
7 The drafted VfM plan should be revised to establish a more robust methodology and
Completed. See above. A two day workshop was conducted with State Team Leaders and Technical
3
approach for identifying the economic framework and assessment plan, which serves as the foundation for any strengthened VfM assessment. Workshops should be conducted to ensure that all MNCH2 staff have an understanding of VfM and how economic tools will be applied and guide the assessment of VfM throughout the course of the programme.
Leads to inform development of the revised vfm strategy. Subsquently, all MNCH2 staff were trained on MNCH2 approaches to measure and ensure VfM.
8 The reported proportion of expenses by resource and programme output categories could be further strengthened and translated to lower levels of the implementing organization (and within the consortium) for enhanced understanding and a complete organizational approach to enhanced value for money or cost-effective programme implementation.
Ongoing. Finance reports now include expenditure breakdown by output, and within each output the proportion of personnel costs compared to reimbursable costs is described. Reports are reviewed at Consortium level, and jointly between DFID and MNCH2 at quarterly Programme Management Meetings.
9 MNCH2 should also continue to improve monthly forecasting to ensure accuracy and to comply with DFID requirements (maximum variance 5% for monthly forecast and within 2% by end of year). This priority should also guide VfM implementation and monitoring over the course of MNCH2.
Ongoing. Forecasting has shown significant improvement and is now included as a key performance indicator within a revised milestone payment plan.
10 The planned political economy analysis should include an assessment of the risk posed by government financial constraints on MNCH2’s ability to achieve results and ensure sustainability. Strategies to mitigate against these risks should be developed jointly with governance programmes, and reference in MNCH2’s risk matrix.
Completed. MNCH2 is engaging with the new DFID funded governance programme (PERL) to manage risk. MNCH2 risk register updated to include measures to manage risk, including government financial constraints.
11 As part of the logframe revision, the risk rating for each output should be reviewed and revised if appropriate, taking into account the changing context of programme implementation.
Completed. Revised risk register developed jointly between DFID and MNCH2, aligning with DFID’s updated risk management framework. Risk ratings for each logframe output amended in accordance with ratings in risk register. Overall risk for project now ‘medium’ compared to ‘low’ previously.
12 MNCH2 should develop a contingency/business continuity plan for fragile states (particularly Yobe) and the challenges of the programme operating effectively in the face of insurgency.
Completed. Contingency plan developed setting out logistics, communication and coordination plans in the event of an immediate threat, and contingency arrangements to ensure delivery against each output in case of increasing insurgency.
13 DFID should pro-actively manage procurement needs of the programme. Procurement timelines should be set out and monitored through DFID’s delivery plan for MNCH2. Quarterly tripartite meetings should be held between DFID, MNCH2 and Crown Agents for information sharing and joint planning.
Ongoing. Procurement plans developed for 16/17 and life of programme. Contract agreement entered with CANL for FY 16/17 procurement requirements, and actively monitered by DFID and MNCH2. Relationship established between MNCH2 and DFID’s new procurement supplier (DPSA) to plan procurement for 17/18 and beyond.
14 DFID should consider renegotiating the contract and milestone payment plan to set out milestones that demonstrate more clearly the increased ambition of the programme over the 5 year period, becoming more challenging and with a greater emphasis on outcome level results towards the outer years as the programme seeks to establish government scale up and sustainability.
Partially completed. Revised milestone payment plan agreed, contract amendment in process. DFID to review draft proposal and issue contract to Palladium for signature.
15 MNCH2 programme should commit to improving the quality and robustness of data reporting, analysis and utilization specifically:
Partially completed. Programme Indicator Reference Sheets completed for all Impact, Outcome and Output indicators. PIRS
4
• MNCH2 should revise the indicator reference sheet of its M&E plan to ensure that each programme data element and indicator is clearly defined.
• MNCH2 should prioritise the development of its programme database to enable the programme to improve its ability to analyse and utilise data for decision-making.
• MNCH2 should bolster the use of innovative analytical products including data visualization to institutionalize the use of data for decision-making.
provide clear definition of indicator, data source, baselines and method of calculation of result. A programme database is under development with supporting database tools. A dashboard is being prepared to ensure that data is captured on time and validated through the programme’s M&E system. However, further work is needed to bring together, review and analyse all data sources, routine results reporting and all data generated by programme tools – see Recommendations 2 and 3 below.
Summary of recommendations for the next year R1: MNCH2 should develop a sustainability plan and exit strategy to ensure that successful interventions are fully embedded within government systems and that government is adequately prepared for programme closure. R2: MNCH2 should bring together, review and regularly analyse all data sources, routine results reporting and all data generated by programme tools. The MNCH2 Communications Strategy should be reviewed and accelerated to ensure the effective dissemination of results and evidence, including through platforms such as the programme website, social media and other local opportunities (eg Nigerian HealthWatch) R3: MNCH2 should facilitate and support the sharing of good practice and cross-learning in supported states and at federal level. MNCH2 technical leads and coordinators should actively generate success stories/lessons learned and report these on a quarterly basis for review and dissemination. R4: MNCH2 should continue to strengthen engagement at Federal level, including support for implementation of the National Health Act and for the development and implementation of health financing and human resources for health policies and strategies. R5. MNCH2 should work with States and at Federal level to develop costing models for primary health care. Information should be shared with and understood by facility managers, and used for advocacy, strategic planning and budgeting. R6. MNCH2 should support states to identify and implement innovative and sustainable health financing models, eg Results Based Financing, health insurance and others. R7. MNCH2 should support implementation of the task shifting policy, to increase access to services and improve efficiency in the health sector. R8. MNCH2 should work to strengthen the Drug Revolving Fund system, rolling out best practice across states, and disseminating lessons learned. R9. MNCH2 should continue to support the FMoH and NPHCDA to ensure that the ISS tools and integrated training manuals are finalized as a matter of urgency. R10. DFID and MNCH2 should set annual targets for all indicators where baselines were established during the reporting year. R11. Given the successful scale up of interventions under Output 2 including the implementation of more effective approaches to attain results, DFID and MNCH2 should jointly review Output 2 indicators targets to ensure these maintain an appropriate level of ambition for the remaining programme years.
5
Consideration should be given to revising targets upwards or, if more appropriate, to reinvesting resources to other outputs. R12. MNCH2 should continue to support implementation of the Human Resource Information System, providing technical input at both state and federal level to ensure a unified national system. Consideration should be given to including heatlh training institutes and student health workers within HRIS, to capture the pipeline of health workers and facilitate HR planning. MNCH2 should engage with W4H in this regard to understand data availability and data needs within health training institutes.
R13. Following review and validation of the OCAT assessments, MNCH2 should further examine differences between states and LGAs in both baseline scores and progress made to understand why some states/LGAs appear to be making better progress than others against OCAT domains.
R14. MNCH2 and DFID should continue to advocate for the establishment of functioning State Primary Health Care Agencies in supported States.
R15. DFID and MNCH2 should jointly determine the level of support to be provided by MNCH2 to the Nahuche Health and Demographic Surveillance Site. A decision should be reached by Dec 16 on whether MNCH2 support should continue. DFID should engage with stakeholders to manage expectations and to mitigate against reputational risk to MNCH2 and DFID, should a decision be reached to end MNCH2 support. R16. DFID and MNCH2 should continue to manage and monitor monthly spend, aiming to ensure at least 80% spend by December. Every month MNCH2 should review and revise the programme FY forecast and monthly spend profile, identifying areas where economy or efficiency savings have been made and bringing this to DFID’s attention. DFID, MNCH2 and Crown Agents should actively manage the procurement of drugs and equipment (valued at £9.3m for FY 16/17), to ensure 100% spend by December. R17. DFID and MNCH2 should actively manage procurement requirements through regular meetings with Crown Agents and with the new procurement supplier, using agreed procurement plans and timelines as a tracking tool to measure progress. R18. DFID should urgently finalize the agreed contract amendement and share with Palladium for signature.
6
A. Introduction and Context (1 page)
DevTracker Link to Business Case: 4526747 DevTracker Link to Log frame: 5567091
Outline of the programme MNCH2 is a 5 year project aiming to improve maternal, newborn and child health and routine immunisation across six States in Northern Nigeria: Kano, Kaduna, Katsina, Zamfara, Jigawa and Yobe. The programme will deliver essential health care for pregnant women, newborns and children and will improve health system coordination through health sector planning and financing and increased demand for and access to high quality health services. Impact measures of success include:
1. Reduction in maternal mortality ratio from 545 to 257 per 100,000 live births 2. Reduction in under 5 mortality rate from 158 to 137 per 1,000 live births 3. Reduction in neonatal mortality rate from 40 to 21 per 1,000 live births
Outcome level results (cumulative) include:
1. 1.4 million safe births attended by a skilled birth attendant 2. 3.9 million women making at least 4 antenatal care visits 3. 700,000 women and newborns receiving a post natal check within 24 hours of delivery 4. 4.9 million children fully immunized 5. 1.2m new acceptors of family planning
Note: Following a review of baselines and and revised methodology for measurement, outcome results have been revised upwards compared to the Business Case targets. Family planning has also been elevated to outcome level rather than output level. Context in which the programme is delivered
Nigeria has the highest number of maternal and child deaths in sub-Saharan Africa, with an estimated 40,000 maternal deaths and 827,000 child deaths each year1,2. The MDG targets for maternal and child mortality were not met. An estimated one in thirty women in Nigeria will have a death related to pregnancy or childbearing and an estimated one in eight children will not survive to their fifth birthday1. With approximately 60% of the nation’s population and having the worst service indicators, Northern Nigeria represents the country’s largest disease burden in terms of maternal and child deaths. Northern Nigeria has the highest child mortality rate and the highest fertility rate, the lowest proportion of children fully immunized and lowest proportion of births attended by a skilled attendant. DFID policy and programming context Reducing maternal, newborn and child deaths is a key priority for the UK Government. DFID’s 2011-15 operational plan committed to saving the lives of at least 50,000 women during pregnancy and childbirth and 250,000 newborn babies by 2015, alongside enabling at least 2 million women to give birth safely and providing access to modern methods of family planning for an additional 10 million women. DFID Nigeria makes a significant contribution to these goals. DFID has been investing in maternal and child health in Nigeria since 2004. Notable successes include strengthening the governance of primary heatlh care, improving quality of care, establishing sustainable drug supply and emergency transport systems and increasing demand for and access to quality services. However Nigeria’s health system remains plagued by fragmentation, poor governance and funding and quality problems – particularly at the primary
1 Trends in maternal mortality: 1990 to 2013. Estimates by WHO, UNICEF, UNFPA, The World Bank and the United Nations
Population Division. Accessible at: 2 Levels and Trends in Child Mortality. Report 2013. Developed by the UN Inter-agency Group for Child Mortality
Estimation. Accessible at: www.childinfo.org
7
healthcare level. The MNCH2 programme was designed to build upon the successes of previous DFID programmes, with a specific focus on building capacity and ensuring sustainability in the government response.
B: PERFORMANCE AND CONCLUSIONS (1-2 pages) Annual outcome assessment The programme is on track to attain the desired outcome “To improve access to and delivery of MNCH and RI services”. As shown in Table 2 below, the programme has exceeded 4 out of 5 outcome targets and marginally underachieved (93%) one target. Table 2: Achieved progress by the MNCH2 programme in Year 2.
Outcome Year 2 target Year 2 achieved (% of target)
Data Source
1. Cumulative number of births in supported states assisted by skilled personnel
326,341 370,414 (114%)
HMIS
2. Cumulative number of pregnant women making at least 4 ANC visits
1,465,745 1,367,149 (93%)
HMIS
3. Cumulative number of newborns and mothers who received care within 24 hours of delivery by a skilled health care provider
134,410 137,716 (102%)
HMIS
4. Cumulative number of children fully immunised against vaccine-preventable diseases
1,492,060 1,602,671 (107%)
HMIS
5. Cumulative number of new acceptors of modern family planning methods
410,888 474,113 (115%)
HMIS
Overall output score and description – A outputs met expectation Table 3: Overall programme output scores
Output Weighting (%)
Score
Output 1: Supported health facilities providing quality integrated RMNCH services
30 A
Output 2: Increased demand for quality integrated RMNCH services in supported states/LGAs
20 A+
Output 3: Improved evidence based decision making and accountability at all levels for the delivery of quality integrated RMNCH services in supported states/LGAs
20 A
Output 4: Supported state/LGA health departments plan and manage resources to deliver quality integrated RMNCH services
20 A
Output 5: Operational research informes MNCH2 programme delivery and national and global RMNCH policy and practice
10 A
8
Key lessons MNCH2 has successfully built upon the achievements of predecessor programmes, and has introduced new interventions to further enhance state and federal capacity for the planning and delivery of primary health care. Evidence to date suggests that both Federal and State governments are receptive to MNCH2 interventions and are willing to replicate and scale-up across programme states and more widely. However, lessons learned from predecessor programmes illustrate that early effort is required to develop programme exit strategies and to ensure government ‘buy in’ and adequate preparation for programme closure. MNCH2 has generated a significant amount of data, including routine results and data generated by programme tools (eg quality of care assessments, capacity assessments, community scorecards). While these data are used for programme monitoring and reporting, the integration of evidence across programme components could be stronger. Further effort is needed to effectively disseminate data, results and evidence and to facilitate the sharing of good practice and cross-learning both within supported states and at federal level (eg learning visits between strong and weak facility health committees). An enabling policy environment at both Federal and State levels is critical to the long-term success of MNCH2 and the ability of the programme to achieve the desired impact (maternal and child lives saved). MNCH2 support for policy formulation and implementation was relatively weak in the first year of the project. Over the past year MNCH2 has made concerted effort to step up engagement at Federal level, playing an active role in the development of national policy and providing support for implementation. As the PATHS2 programme draws to a close, MNCH2 engagement in this space is even more critical, particularly to support implementation of the National Health Act, health financing policies and strategies, and human resources for health policies and strategies. These roles should occur alongside the strong technical input provided by MNCH2 to date in areas such as the development of guidelines and training tools and MNCH2 participation in specific technical groups on, for example, immunisation, RMNCH etc. Recommendations: R1: MNCH2 should develop a sustainability plan and exit strategy to ensure that successful interventions are fully embedded within government systems and that government is adequately prepared for programme closure. R2: MNCH2 should bring together, review and regularly analyse all data sources, routine results reporting and all data generated by programme tools. The MNCH2 Communications Strategy should be reviewed and accelerated to ensure the effective dissemination of results and evidence, including through platforms such as the programme website, social media and other local opportunities (eg Nigerian HealthWatch) R3: MNCH2 should facilitate and support the sharing of good practice and cross-learning in supported states and at federal level. MNCH2 technical leads and coordinators should actively generate success stories/lessons learned and report these on a quarterly basis for review and dissemination. R4: MNCH2 should continue to strengthen engagement at Federal level, including support for implementation of the National Health Act and for the development and implementation of health financing and human resources for health policies and strategies. Has the logframe been updated since the last review? Yes. As recommended at the last annual review, logframe revisions were made to align outputs more closely with the programme theory of change and to better capture programme interventions. Indicators were also added to measure quality of care.
9
Following a review of outcome target baselines and a revised methodology for measurement, outcome results were revised upwards compared to the Business Case targets. Family planning has also been elevated to outcome level rather than output level. Risk ratings for each output were reviewed, and some revised upwards bring the overall programme risk rating to ‘medium’
C: DETAILED OUTPUT SCORING (1 page per output)
Output Title Supported health facilities providing quality integrated RMNCH services
Output number per LF 1 Output Score A
Risk: Major Impact weighting (%): 30%
Risk revised since last AR? Yes (prev med)
Impact weighting % revised since last AR?
No
Indicator(s) Milestones Progress Data Source 1.1: Cumulative number (percentage) of supported health facilities with at least 1 health worker trained on integrated RMNCH service delivery through MNCH2 support
458 (48%)
487 (51%) MNCH2 training reports, health facility surveys
1.2: Percentage of women accessing RMNCH services in supported health facilities reporting satisfaction with services
Establish baseline Baseline established at 66%
Client exit interview
1.3: Number (percentage) of supported health facilities with minimum stock of essential RMNCH commodities during the past three months
Establish baseline Not completed. Activity was scheduled for Q4 FY 15/16, but due to budget restraints imposed by DFID at the end of the financial year, activity deferred to Q1 FY 16/17
LIAS
1.4: Percentage of supported health facilities meeting minimum quality standards according to the MNCH2 quality of care tool
Establish baseline
Basline established at 44%
QoC assessment tool
Key Points In Year 2 of the programme the selection of health facilities for MNCH2 support was finalized, in consultation with State Governments. A total of 954 facilities across all 6 States were selected. These will make a signification contribution to the government’s vision of ‘One PHC per ward’, and establishment of 10,000 functioning PHCs by 2018. However, a fully function PHC requires the availability of basic emergency obstetric services 24 hours a day, 7 days a week. Many facilities do not provide such a service, and the human resource requirements and costings for 24 hour provision have not been fully worked out. MNCH2 support to facilities includes health worker training on RMNCH services, introduction of and training on Standard Operating Procedures/job aids, the provison of medical equipment, medicines and consumables and rehabilitation/upgrade of facilities. MNCH2 has also collaborated with the DFID funded Solar programme in Kaduna State to ensure that MNCH2 facilities are among those selected for solar equipment to ensure the maximum benefit of both programmes.
10
During the reporting period MNCH2 continued to roll out the integrated MNCH training package. This was complemented by post training integrated supportive supervision (ISS) to further assess provider competence and to provide mentoring and coaching for further improvements. MNCH2 has provided technical support to the National Primary Healthcare Development Agency (NPHCDA) to finalize the national integrated training manual for primary health care workers, and to the FMOH to develop a national Integrated Supportive Supervision tool. Technical input was also provided for development of the national task shifting policy. At the start of the reporting year a Client Exit Survey was applied to assess women’s satisfaction with RMNCH services. However, the results showed very high levels of satisfaction which did not seem consistent with observed weaknesses in health care provision (eg inadequate drugs, equipment, poor quality facilities). This finding is in keeping with global evidence which suggests that patients often report high levels of satisfaction upon the immediate completion of treatment if they receive the outcome they expect (ie received drugs, delivered safely), even if their experience of care was poor. Hence MNCH2 revised the approach for assessing client satisfaction. This is now measured by including questions on client satisfaction within the Community Scorecard, which is measured at community level rather than to clients immediately upon exit from the health facility. Overall, 66% of women who accessed RMNCH services reported satisfaction with the services received. To further validate the findings, qualitative research through focus group discussions is planned for the next reporting year. To ensure the sustainable supply of medicines and consummables, MNCH2 has worked to strengthen state, LGA and facility supply chain management, in particular to ensure capacity to manage the various vertical supply chains currently in operation (eg vaccines, family planning commodities, MNCH drugs). This has included the development of a pool of State and LGA ‘master trainers’ on Integrated Logistics and Supply Chain Management. Step down training to facility staff has been initiated. MNCH2 facilitated the establishment of Logistics Working Groups in 3 States where these did not previously exist (Kano, JIgawa and Katsina) and reviewed existing Drug Revolving Fund guidelines to further enhance supply chain coordination and quality assurance of medicines. Medical equipment, drugs and consummables ordered by MNCH2 in Year 1 of the project were delivered during the past year and distributed through State Medical Stores to facilities. A three year quantification plan for medicines and consumables for project years 3 - 5 was finalized with each State, setting out the contribution that MNCH2 will make to capacitate facilities and central medical stores, alongside strengthening supply chain management. Following DFID approval, MNCH2 is now working with DFID, Crown Agents and the new procurement supplier to ensure delivery of planned procurements for the remainder of the project. To measure the number and percentage of supported facilities with minimum stock of essential RMNCH commodities, MNCH2 planned to conduct a logistics assessment using the Logistics Indicators Assessment Tool, a quantitative data collection instrument developed by the USAID DELIVER Project. This is a holistic tool that not only measures stock availability, but also assesses logistic management processes to identify weaknesses and areas for improvement. The LIAT assessment was due to take place in the final quarter of the reporting year. However, in Feb 2016, DFID informed MNCH2 of a reduction in available funding for Q4 and hence, in agreement with DFID, the LIAT assessment was deferred to take place in the first quarter of Year 3. MNCH2 has developed and comprehensive quality of care assessment tool (QoCAT. This is administered by survey at facility level and measures quality of care across a number of domains including:
• ability to provide specific MNCH services
• availability of skilled health works
• availability of functioning medical equipment
• quality of infrastructure, including availability of water and electricity
• drug and consumable availability
• use of guidelines/SOPs
11
• enagement with community (eg outreach services, referral systems, participation of facility head in Ward Development Committee)
• patient information available (eg service lists, costs, IEC materials)
• use of Health Management Information Systems
• respect and cognition – eg availability of female workers, patient privacy, counselling provided A baseline assessment was conducted in Year 2 which showed that 44% of facilities met minimum quality of care standards (ie composite QoC score of 60% or above). Following each assessment facility specific action plans were developed. Recommendations R5. MNCH2 should work with States and at Federal level to develop costing models for primary health care. Information should be shared with and understood by facility managers, and used for advocacy, strategic planning and budgeting. R6. MNCH2 should support states to identify and implement innovative and sustainable health financing models, eg Results Based Financing, health insurance and others. R7. MNCH2 should support implementation of the task shifting policy, to increase access to services and improve efficiency in the health sector. R8. MNCH2 should work to strengthen the Drug Revolving Fund system, rolling out best practice across states, and disseminating lessons learned. R9. MNCH2 should continue to support the FMoH and NPHCDA to ensure that the ISS tools and integrated training manuals are finalized as a matter of urgency. R10. DFID and MNCH2 should set annual targets for all indicators where baselines were established during the reporting year.
12
Output Title Increased demand for quality integrated RMNCH services in supported states/LGAs
Output number per LF 2 Output Score A+
Risk: Minor Impact weighting (%): 20%
Risk revised since last AR? No
Impact weighting % revised since last AR?
No
Indicator(s) Milestones Progress Data source
2.1: Cumulative number of communities reached with small group interventions (through male groups, religious leaders and TBAs) to increase demand for, and uptake of RMNCH services
1,683 2,766 Programme demand creation monitoring reports
2.2: Cumulative number of young women benefiting from safe space interventions
17,301 22,318 YWSG monthly summary sheet
2.3: Cumulative number of women with pregnancy-related conditions transported through the Emergency Transport Scheme (ETS) in supported communities
12,874
16,213 ETS log sheet summary
Key Points Activities to increase demand for RMNCH services were scaled up during Year 2, with active learning on approaches taken and adapatations where relevant.
The initial community outreach package developed by MNCH2 included engagement with traditional and religious leaders alongside the establishment of young women’s support groups. However a literature review conducted by MNCH2 identified that men, and in particular husbands, play significant roles in women’s utilization of health services. As a result, MNCH2 adapted the community engagement strategy to actively target husbands and fathers in addition to religious and traditional leaders (see Output 5).
The Young Women’s Support Groups were expanded, and the frequency of sessions condensed to increase completion rates. To date, over 22,000 young women have completed at least 3 modules of the Safe Space Initaitive (SSI) training manual, covering personal and environmental hygiene, nutrition, ANC, danger signs of pregnancy, PNC, immunization and child welfare. Following MNCH2 review of the scheme, gaps were identified related to healthy timing and spacing of pregnancy. As a result the SSI manual was updated in consultation with representatives from the Fedearl Ministry for Women’s Affairs, Federal Ministry of Health, and Federation of Muslim Women Organisation of Nigeria.
A total of 182 volunteer Emergency Transport Scheme (ETS) drivers were trained in Year 2 and quarterly review meetings were held with the National Union of Road Transport Workers to discuss and share experiences with ETS implementation, including how to get more drivers involved in the ETS and to develop sustainability strategies. A review of the Emergency Transport Scheme identified that traditional birth attendants (TBAs) play a crucial role to link women to ETS drivers. As a result MNCH2 has scaled up training/mobilization of TBAs to make referrals and link women and children with the ETS.
As a result of the flexible strategies adopted all targets under Output 2 were exceeded. Recommendations R11. Given the successful scale up of interventions under Output 2 including the implementation of more effective approaches to attain results, DFID and MNCH2 should jointly review Output 2 indicators targets to ensure these maintain an appropriate level of ambition for the remaining programme years.
13
Consideration should be given to revising targets upwards or, if more appropriate, to reinvesting resources to other outputs.
Output Title Improved evidence based decision making and accountability at all levels for the delivery of quality integrated RMNCH services in supported states/LGAs
Output number per LF 3 Output Score A
Risk: Minor Impact weighting (%): 20%
Risk revised since last AR? No
Impact weighting % revised since last AR?
Yes (prev 15%)
Indicator(s) Milestones Progress Data source
3.1: Percentage of health facilities in supported States whose routine HMIS returns meet minimum national requirement for data quality standard
63%
67% Programme Data Quality Assessment Reports
3.2: Percentage of MDR committees at secondary facilities in supported States/LGAs using MDR findings to improve quality of integrated RMNCH services at all levels
20%
86% (113 facilities) MDR reports
3.3: Number (percentage) of FHCs established in supported primary health care facilities.
233 (29%)
264 (33%) FHC self-assessment reports
3.4: Number (percentage) of FHCs in supported primary health care facilities meeting good standards against all three FHC key roles
186 (80% of 233)
216 (93% of 233)
FHC self-assessment reports
3.5 Number of States that demonstrate use of RMNCH data to inform the development of health sector plans
3/6 (with minimum 80% score)
4/6 (Jigawa, Katsina, Kaduna, Kano)
Key informant interviews to assess use of evidence
Key Points With MNCH2 support, States have made good progress towards improving the collection and use of data to inform plans, activites and resource allocations. MNCH2 has supported states to conduct Data Quality Assessments of Health Management Information System (HMIS) returns. These assessments were conducted on facilities directly supported by MNCH2 and on facilities that did not receive direct MNCH2 support (as a proxy measure of the State’s ability to disseminate best practice and cascade tools and processes introduced by MNCH2). The proportion of health facilities whose HMIS returns met minimum requirements was higher than target for the review period. However, in order to ensure adequate HMIS returns, it was necessary for MNCH2 to copy and distribute HMIS materials including patient registers (eg over 2,700 ANC registers) and monthly and quarterly summary forms. While this was necessary in the short term, MNCH2 will need to ensure that federal and state bodies fulfil their responsibilities for the adequate production and distribution of HMIS resources. MNCH2 has also supported the development of a national Human Resources for Health Information System (HRIS). During the past year MNCH2 worked with PATHS2 to transition the HRIS database managed by PATHS2 (over 27,000 HRH entries across 3 states) to MNCH2 and engaged with the FMOH to develop a national system. However further effort is needed to harmonize and achieve a robust, cost effective and sustainable HRIS database that is nationally driven and meets the requirements of all states. MNCH2 has also begun to take over, from PATHS2, support at Federal level for the National Health Data Consultative Meeting (NHDCC) and Technical Working Group, including support for the review of NHDCC Terms of Reference and review of HMIS tools.
14
MNCH2 supported the establishment of 134 Maternal Death Review (MDR) committees and 15 verbal autopsy committees across the six states and roll out of the national Maternal and Perinatal Death Surveillance and Review framework. Findings showed that 422 of maternal death were reported of which 222 were reviewed within an acceptable time. A total of 177 actions were developed and 140 implemented across functional committees. A total of 264 Facilty Health Committees were established or reactivated/supported by MNCH during the reporting year, of which 216 (93%) attained performance standards. Among other roles, FHCs maintain a two-way and open dialogue between communities and service providers on improvements in health services, by obtaining feedback from the community on health service performance and improvements needed. Key information intereviews identified that four out of six states demonstrated the review and use of data to inform state strategic and operational planning. The key sources of data identified by informants included HMIS, ISS reports, QoC reports and community scorecards. Recommendations R12. MNCH2 should continue to support implementation of the Human Resource Information System, providing technical input at both state and federal level to ensure a unified national system. Consideration should be given to including heatlh training institutes and student health workers within HRIS, to capture the pipeline of health workers and facilitate HR planning. MNCH2 should engage with W4H in this regard to understand data availability and data needs within health training institutes.
15
Output Title Supported state/LGA health departments plan and manage resources to deliver quality integrated RMNCH services
Output number per LF 4 Output Score A
Risk: Moderate Impact weighting (%): 20%
Risk revised since last AR? Yes (prev low)
Impact weighting % revised since last AR?
Yes (prev 15%)
Indicator(s) Milestones Progress Data Source
4.1 Average capacity score of supported LGAs in state on organizational governance, coordination, and partnership as measured by Organizational Capacity Assessment Tool (OCAT)
Phase 1 LGAs (n=65)
Average 25% (red range)
Phase 2 LGAs (n=65)
Establish baseline
Target exceeded:
Phase 1 LGAs = 62%
(Range Zamfara 58% to Yobe 75%)
Phase 2 and 3 LGAs (94 in total) = 25%
LGA OCAT
4.2: Average capacity score of supported LGAs in state on organizational planning and budgeting as measured by Organizational Capacity Assessment Tool (OCAT)
Phase 1 LGAs (N=65)
Average 41% (amber range)
Phase 2 LGAs (n=65)
Establish baseline
Target exceeded Phase 1 LGAs = 71% (Range Zamfara 66% to Yobe 85%) Phase 2 and 3 LGAs = 37% (baseline)
LGA OCAT
4.3 Average capacity score of supported LGAs in state on capacity for resource mobilization as measured by Organizational Capacity Assessment Tool (OCAT)
Phase 1 LGAs (n=65)
Average 17% (red range)
Phase 2 LGAs (n=65)
Establish baseline
Target exceeded
Phase 1 LGAs = 47%
(Range Zamfara 24% to Yobe 90%)
Phase 2 and 3 LGAs = 1% (baseline)
LGA OCAT
4.4: Average capacity score of state PHCDAs/PHCMBs/GHSB as measured by Organizational Capacity Assessment Tool (OCAT)
Establish baseline Baseline established Average = 49% (amber range)
State OCAT
Key Points In Year 2, MNCH2 provided training and supervision to States and LGAs in governance and management, budgeting and planning and resource mobilisation. Specific examples include: - Establishment/revitilization of MNCH steering committees at LGA level as a governance structure to
oversee, monitor and coordinate the health system for MNCH services - Strengthened capacity of State/LGA health and planning officers on planning and development of
Medium Term Sector Strategic (MTSS) and Annual Operational Plans (AOP). All 6 States have an MTSS and AOP and all LGAs have an AOP
- Supporting the development of state proposals to access a federal grant ($1.5m) for MNCH services under the Saving One Million Lives, World Bank Payment by Results programme
Organisation Capacity Assessments of States and LGAs were conducted using a validated Organisational Capacity Assessment Tool (OCAT). Targeted capacity development plans were drafted to address weaknesses identified. These plans are aligned with the LGA/State Annual Operational Plans. All LGAs
16
have responded positively to the introduction of OCAT, recognizing it as a useful and easy way of measuring capacity over a period of time, and tool to support performance improvements. All Phase 1 LGAs (65 LGAs across 6 states) have shown marked improvements in their capacity during the past year. However, the OCAT is a self-assessment tool and it is possible that a degree of over-achievement has been reported. The first (baseline) assessments are conducted with extensive MNCH2 support as a capacity building exercise in its own right. Thereafter the OCAT is led by the LGA/State. It is possible that assessments may have been reported optimistically. In the coming year MNCH2 will review the process and validity of the assessments, refining guidance materials as necessary, aiming to reduce optimism bias for future assessments. None-the-less, given the low performance identified in baseline assessments it is likely that rapid development has been made in some domains as quick wins are realized. The weakest domain identified by OCAT at LGA level is resource mobilisation. Baseline assessments identified that LGAs have limited or no knowledge or skills in resource mobilisation, no strategy in place to address this nor any ‘traditional’ approaches to resource mobilisation to lack of understanding of the process required. However large improvements in this domain have been demonstrated in Phase 1 LGAs, primarily due to the establishment of MNCH steering committees as a ‘home’ for resource mobilisation activities. Examples of resources mobilized by MNCH steering committees include:
• Drugs were obtained by Maiadua LGA MNCH steering committee from the Nigeria Custom Service. The drugs are estimated to last for one a year. Additionally, Sandamu LGA received 100,000 tablets of iron/folic acid supplement for the PHC clinic in Fago
• Commitment by a local plastic manufacturing company to improve a health facility in Gezawa LGA following MNCH Steering Committee advocacy;
• Renovation of health facilities in Kumbotso LGA and Sandama LGA through resources secured by the MNCH steering committee
At State level, fragementation in the management of PHC services remains a significant challenge. Despite effort by MNCH2 and predecessor programmes states are very slow to adapt the ‘PHC under one roof’ model and make operational changes to the structure and systems, despite the OCA baseline assessment that has identified this as a weakness. However some positive changes have been supported:
• Dismantling of the Gunduma Health System Board in Jigawa and passing a law for the establishment of Primary Health Care Development Agency for achievement of Primary Health Care Under One Roof;
• The signing into law of the Kaduna State Primary Health Care Development Agency and appointment of Executive Secretary and Governing Board
• Zamfara has had the PHC law withdrawn for amendments and efforts by MNCH2 to advance this resulted in its membership of an advisory committee to the Governor.
• Katsina State has sent the amended draft law for passing by the House of Assembly In Kano State, although legislation for the establishment of a State PHC Development Board has been passed, the SPHCDB Governing Board has not obeen appointed and the SPHCDB does not receive autonomous funding.
Recommendations
R13. Following review and validation of the OCAT assessments, MNCH2 should further examine differences between states and LGAs in both baseline scores and progress made to understand why some states/LGAs appear to be making better progress than others against OCAT domains.
R14. MNCH2 and DFID should continue to advocate for the establishment of functioning State Primary Health Care Agencies in supported States.
17
Output Title Operational research informes MNCH2 programme delivery and national and global RMNCH policy and practice
Output number per LF 5 Output Score A
Risk: Minor Impact weighting (%): 10%
Risk revised since last AR? No Impact weighting % revised since last AR?
No
Indicator Milestone Progress Source
5.1: Number of OR studies conducted and disseminated by the programme 4
3 (note target not met due to budget restraint imposed by DFID at end of financial year)
Research reports
5.2: Number of instances where evidence generated by the programme leads to changes that aim to improve efficiency and/or effectiveness of MNCH2 implementation
2
2 Research and programme reports
5.3: Number of instances where evidence generated by the MNCH2 programme is presented at the national level to inform national RMNCH policy and practice
2 0 (note target not met due to budget restraint imposed by DFID at end of financial year)
Research and programme reports
5.4: Number of publications developed and disseminated by the programme 1
2 Programme reports
Key Points
In Year 2 MNCH2 consolidated capacity building for State Operations Research Committees (ORACs) and subcommittees in all six MNCH2 states. A series of state and national OR workshops were conducted, including with government stakeholders, to generate research questions. Four operations research studies were planned and three were completed. MNCH2 could not undertake the fourth study due to DFID imposing a budget restriction in the final quarter of the reporting year. MNCH2 has also supported the State Government of Zamfara to develop a costed strategy for the Nahuche Health and Demographic Centre. Indicator 5.1 MNCH2 successfully completed the following Operations Research Studies:
1. Assessment of Emergency Transport Systems in Kaduna and Jigawa states This study concluded that implemenenting ETS alongside other demand creation activities is more effective than either intervention in isolation and that traditional birth attendants (TBAs), who received training in referals from MNCH2, are central to the utility of ETS to transport women to the health facility. A need to strengthen links between facilities and ETS, including means of identification of ETS drivers was identified. 2. Assessment of facility health committees (FHCs) in improving quality of care in health facilities in Kano,
Kaduna and Jigawa states This study concluded that FHCs are perceived to contribute to quality of MNCH services especially in the area of maintenance of equipment and infrastructure, drugs and commodity supplies and resource mobilization, and are perceived to positively increase awareness and mobize communities for utilization
18
of services. However FHC members’ require further capacity building in the area of work planning, resource mobilization and documentation need further strengthening.
3. Influencing demand for maternal and child health services in northern Nigeria: a review of the literature This study concluded that men play significant roles in women’s utilization of healthcare services for themselves and their children: programmes need to invest in men’s groups especially husbands. Women’s decision to seek health services for themselves and their children is influenced by their perceptions of the opinions and behaviours of those around them, including, traditional leaders, religious leaders and scholars, health providers, traditional healers, family, friends, and neighbours Indicator 5.2 Number of instances where evidence generated by the programme leads to changes in MNCH2 implementation MNCH2 has adopted new/revised strategies based on the findings of the above research studies. Specifically, TBAs are being more actively included in demand creation activities and ETS schemes and related training since their impact is greater than any other group. The community mobilization package now includes specific men’s groups to target husbands and fathers, in addition to the existing work with women, and with traditional and religious leaders. Indicator 5.3 Number of instances where evidence generated by the MNCH2 programme is presented at the national level to inform national RMNCH policy and practice MNCH2 had planned the dissemination of the evidence generated by the programme at the federal and state levels in quarter 4. However, these could not be achieved due to a budget restriction imposed by DFID in Q4. Indicator 5.4 Number of publications developed and disseminated by the programme Two research studies were submitted for publication during the review year. The first “Accountability for quality of care: Monitoring all aspects of quality across - a framework adapted for action” was published in the peer reviewed journal, International Journal of Gynecology and Obstetrics. MNCH2 submitted the literature review study “Influencing demand for maternal and child health services in northern Nigeria: a review of the literature” to the journal, Women’s Health International. Recommendations R15. DFID and MNCH2 should jointly determine the level of support to be provided by MNCH2 to the Nahuche Health and Demographic Surveillance Site. A decision should be reached by Dec 16 on whether MNCH2 support should continue. DFID should engage with stakeholders to manage expectations and to mitigate against reputational risk to MNCH2 and DFID, should a decision be reached to end MNCH2 support.
19
D: VALUE FOR MONEY & FINANCIAL PERFORMANCE (1 page) Key cost drivers and performance Costs are similar to those described in the business case and include include human resources, travel, training, rehabilitation of clinics, and expenses for supplies and office operating expenses. Additional costs include the procurement of drugs, commodities and equipment, managed through a separate contract between DFID and the procurement agent. As demonstrated in Table 4 below, MNCH2 has strived to minimize costs and ensure efficiency savings across programme delivery. VfM performance compared to the original VfM proposition in the business case The Business Case did not set out specific vfm measures for the programme.
In response to the recommendations of the last annual review, MNCH2 developed a vfm strategy, setting out a framework for assessing vfm throughout the programme cycle and a set of economic questions and indicators. A workshop was held with MNCH2 State Team Leaders and Technical Leads to provide input to the vfm strategy, and all staff have been trained on the vfm framework and indicators.
Overall, MNCH2 has shown good progress on the vfm indicators with 5 out of 7 indicators measured to date showing favourable performance compared to benchmark or trends. Table 4 presents vfm indicator results for the programme as a whole. For most indicators state level data has also been collated allowing the programme to compare performance between states to learn from and replicate best practice.
Table 4 Performance Against VFM Indicators
Indicator Performance Benchmark/Trend & Analysis
Economy
Total savings as % of total expenditure for period
Year 2 savings approximately N7.5m
Big ticket’ savings include: - Negotiation on facility assessment subcontract - Sharing Palladium seucrity office space with TSU project - ETS state master trainers used instead of national master trainers - Reduced daily payment rate for trainers
Management and Administration costs as % of total expenditure
Cumulative Yr 2 = 39% Cumulative Y 3 = 22%
Overall the project budget for administration costs is 17%. Initial start up costs were high, but the downward trends indicates that the project is on track to meet, or fall under the project admin cost target
Average unit cost of fees
MNCH2 = £171 (long term 90% and short term 10%) PATHS2 as comparator long term fees only (2014) = £186
MNCH2 fee rates compare favourably with PATHS2, a similar programme operating in 4 of the MNCH2 programme states
Efficiency
Unit cost of participant day for MNCH2 training, meetings and events
Measured for 4 states: Kano = £25.25 average Yobe = £58.94 average Zamfara = £32.73 average Jigawa = £63.74 average
Costs vary significantly from state to state, but are depending on the type of training, number of days and type of participant. MNCH2 continues to monitor training costs as state level to ensure consistency and identify where savings can be made.
Unit cost of training a health worker on integrated RMNCH services
MNCH2 = £386 for integrated training package PRRINN-MNCH ranged from £123 to £941 for separate training components PATHS2 = £625 for advance life skills training in Kano
MNCH2 offers an integrated training package to minimize the number of days and frequency with which a health worker is absent form the workplace. The unit costs for the whole integrated package is significantly lower than the cost of specific modules offered by comparator programmes. While costs are lower, it is not possible to directly compare the learning outcomes of the different programmes, ie a shorter and cheaper training may not provide the same quality.
20
However, MNCH2 through its Quality of Care Strategy is monitoring health worker performance, facility attendance rates and client satisfaction while will provide proxy measures to ensure that the health worker training package is adequate.
Unit cost of supporting a woman with pregnacy complications
MNCH2 = £40 (Q1 to Q3 service data) PRRINN-MNCH: £10 - £18 PATHS2 = £21 - £87
Only training and ETS support costs are included (fuel and other running costs are met by the driver/community). MNCH2 ETS service statistics were only available for Q1 –Q3 at the time of calculation, when Q4 figures are included the unit cost is likely to fall. None-the-less costs seem higher than comparator programmes and should continue to be monitored.
Unit cost of reaching a small group/community with health promotion
MNCH2 = £388 (Q1 to Q3 service statistics) PRRINN-MNCH = £381
MNCH2 costs include full year training costs, but only include outreach statistics for Q1 to Q3, hence the unit cost is likely to fall when Q4 data are added. MNCH2 compares favourably to PRRINN-MNCH costs
Effectiveness
Increase in service uptake
See outcome indicator results – all targets exceeded except ANC 4+
MNCH2 remains on track to attain outcome targets overall. Although ANC4 has slightly underachieved (93%) it is too early to determine if this will remain off track, or can be ‘caught up’ in future years. Under the milestone payment mechanisms MNCH2 receives payment when outcome targets are reached. As ANC4 was not reached, no payment has been made against this milestone.
Cost effectiveness
Cost per DALY Not yet measured Business case estimate = £35 per DALY
This indicator will be measured at the next Annual Review as outcome level results were not available at the time of the vfm analysis
Equity
% of CHEWs and nurses trained by MNCH2 who are female
377 out of 460 workers trained = 82%
No target has been set for this indicator. However since a shortage of female health workers is a main barrier to women and children accessing health care it is important for MNCh2 to prioritize training female health workers. With 82% of workers trained to date being female, MNCH2 is adequately addressing this challenge.
Sustainability
Government contributions as % of MNCH2 expenditure
MNCH2 = 13% PATHS2 = 10 – 15% PRRINN-MNCH = upto 50%
This indicator was measured for the first time this year, and data may not be complete. However, as State teams becomeo familiar with the templates the accuracy and completeness of data will improve. The largest government contributoin was in Kano state, where the state government made a significant contribution towards commodities and logistics.
Assessment of whether the programme continues to represent value for money
MNCH2 continues to offer vfm. Programme costs are within budget, and the quantity and quality (where measurable) of outputs and outcomes is high. Vfm indicators demonstrate good performance against benchmarks overall. Quality of financial management In the early months of the reporting period, monthly variance between forecast expenditure and actual expenditure was high (eg 35% variance in May 2015). MNCH2 put in place measures to address this, including greater attention to forecasting and budget monitoring at State level. Variance improved throughout the year (eg 1% in Nov 15) although some inconsistencies remained between months.
21
MNCH2 responded well to DFID’s request to front load expenditure in the months April to December 2015 in order to support attainment of DFID’s ODA target. This included effort to increase programme activities during Q1 – Q3 and scale back in Q4. However there were delays in the procurement of drugs and equipment, and some delays in the completion of facility rehabilitation which meant that planned spend in Q4 was higher than desired. DFID imposed a budget restriction mid-way through Q4, requiring MNCH2 to stop planned activities and to further delay the completion of facility rehabilitation until FY 16/17. MNCH2 managed this budget restriction well, by rescheduling activities into FY 16/17 and preparing a revised FY 16/17 budget for DFID review and approval. Recommendations R16. DFID and MNCH2 should continue to manage and monitor monthly spend, aiming to ensure at least 80% spend by December. Every month MNCH2 should review and revise the programme FY forecast and monthly spend profile, identifying areas where economy or efficiency savings have been made and bringing this to DFID’s attention. DFID, MNCH2 and Crown Agents should actively manage the procurement of drugs and equipment (valued at £9.3m for FY 16/17), to ensure 100% spend by December.
Date of last narrative financial report 30th July 2016 Date of last audited annual statement July 2016
22
E: RISK (½ page)
Overall risk rating: Moderate
Overview of programme risk MNCH2 and DFID actively maintain a risk register that considers risks against 6 categories: programme delivery, context, operational, fiduciary, reputational and safeguarding Mitigating actions have been put in place to address all risks captured in the risk matrix. The risk matrix is reviewed by MNCH2 and DFID at quarterly programme management meetings to assess the adequacy of mitigation measures, and to identify any new risks that have emerged. The main risks at the current time are: 1. State Governments fail to hire, deploy and retain health workers. Mitigating actions include: MNCH2 technical support for a strengthened Human Resource Information System to provide data for human resource decision making and advocacy and accountability; MNCH2 techical support for state budget and planning processes including the allocation of resources for health workers; collaboration with the DFID funded Women for Health programme, which primarily works to increase the pipeline of female health workers; and advocacy to State Governments for health worker deployment by MNCH2 and DFID (including lead adviser, regional coordinators and higher officials where relevant).
2. Failure to attain state wide impact beyond the immediate programme intervention facilities and LGAs
To mitigate against this risk MNCH2 works to build State and Federal capacity to adopt and scale up interventions introduced by the programme. A number interventions are implemented at state level, including HMIS strengthening, HRIS and logistics management systems. Outcome targets have been set at State level to ensure that MNCH2 can demonstrate success across the state as a whole, and not just in supported facilities or LGAs. 3. Increasing government financial constraints reducing health expenditure, and impacting sustainability
of programme. Although identified as a risk, financial constraints have also created opportunity for MNCH2 to support states to use resources more efficiently. Examples include the identification of ghost workers through the HRIS, and efficiency gains in supply chain management. Additional measures to mitigate agains this risk include MNCH2 support to state goverments for the development of medium term health sector strategies and, in collaboration with DFID funded governance programmes, technical support for states to develop annual plans and set realistic budgets, and advocacy by MNCH2 and DFID for budget release where necessary. Outstanding actions from risk assessment There are no outstanding actions from the programme risk assessment. An independent financial audit, covering the period from June 1st 2014 to Dec 31st 2015 was recently conducted, including assessment of the programme’s financial control and asset monitoring processes. The report has been shared with DFID in draft form, but still to be finalized. No significant concerns have been identified in the report.
23
F: COMMERCIAL CONSIDERATIONS (½ page)
Delivery against planned timeframe The project remains on track to achieve desired results within the project period. Delays in the procurement of medical equipment and supplies were experienced in Year 1 and Year 2 of the programme. A number of factors contributed to this including the scale and complexity of procurement requirements and the need to secure State input to procurement quantifications and plans. Additionally, due to Q4 funding constraints DFID instructed MNCH2 and the procurement agent to slow down planned procurement in order to enable payment if financial year 15/16. Lessons have been learned from these experiences. In particular, a three year quantification exercise was completed by MNCH2 and approved by DFID.. A new contract agreement has now been secured with Crown Agents for the procurement of equipment, drugs and consummables for year 3 of the project, and DFID has supported initial engagement between MNCH2 and the new procurement supplier for the the procurement of goods and equipment for Years 4 and 5. In response to the last annual review the MNCH2 contract model was revised. The number of milestones was reduced to focus on a smaller number of higher level milestones, reducing the transactional cost for both MNCHs and DFID, and key performance indicators related to programme reporting and financial management were introduced. However the final contract amendment has not yet been signed. Recommendations: R17. DFID and MNCH2 should actively manage procurement requirements through regular meetings with Crown Agents and with the new procurement supplier, using agreed procurement plans and timelines as a tracking tool to measure progress. R18. DFID should urgently finalize the agreed contract amendement and share with Palladium for signature. Performance of partnership(s) The MNCH2 programme is delivered by a consortium led by Palladium. The programme partnership is functioning effecively, and a programme management manual is in place to guide consortium partners’ oversight of MNCH2. Palladium and the MNCH2 team conduct quarterly meetings with the seven consortium members at which operational issues, partners’ contributions, policy matters, and administrative and financial issues are discussed. The MNHC2 Programme Management Board meets biannually. Monthly meetings take place between Palladium and DFID. The programme participates in DFID coordination meetings in Abuja, and in monthly DFID-led programme management meetings at state level. Asset monitoring and control MNCH2 maintains an asset inventory that is updated annually and share with DFID for review. Spot checks to verify programme assets were carried out as part of the recent independent financial audit. This included spot checks at the MNCH2 head office and state offices. No concerns were identified.
24
G: CONDITIONALITY (½ page)
Update on partnership principles (if relevant) Partnership Principles were not considered within the MNCH2 business case. In September 2014 DFID Nigeria conducted an assessment of the FGN commitment to the Partnership Principles which reconfirmed the UK’s declared policy that that no UK aid money should go directly to the Government of Nigeria as an anti-corruption measure and to avoid substituting for the country’s own resources. The MNCH2 programme is consistent with this policy. As the programme is implemented directly by a service provider and no funding is being directed through Government, the Partnership Principles are not relevant for the direct management and monitoring of this project. Opportunities to leverage political will and buy-in, as well as allocation of government resources (financial, personnel, technical, administrative) are continually being explored and scaled up where possible. For example, a tripartite MoU, between Kaduna State Government, DFID and the Bill and Melinda Gates Foundation (BMGF) has been prepared, outlining the contribution of DFID funded programmes (MNCH2 and PERL), BMGF funded programmes and Kaduna State Government towards strengthening primary healthcare in Kaduna State. The MoU will be signed by all parties in Sept 2016.
H: MONITORING & EVALUATION (½ page)
Evidence and evaluation No evidence has emerged to challenge project design or rationale since approval of the Business Case. Programme monitoring takes place through regular quarterly narrative reports against the project logframe and quarterly financial reports. Monthly meetings are held between DFID and senior programme management. Field visits are conducted throughout the year by DFID (lead adviser and/or regional coordinator), including visits to project sites and meetings with government stakeholders. Annual review process This annual review write up was a desk based exercise drawing on materials from a number of sources:
1. Project documents: Year 3 Annual Report and Quarterly Reports 2. Report of Joint Partner Technical Review – A 12 member team, drawn from across all consortium
partners conducted a programme review in Jun 2016. The team met with key informants and programme beneficiaries including:
o SMOH Kano and Jigawa States o Selected primary and secondary health care facilities o Hospital Management Board Kano o Primary Healthcare Board Kano o Drug Management Supply Agency Kano and Jigawa o Meetings with DFID funded WINN and W4H programmes o Shawu Kangare Community
3. Back to office reports from DFID Regional Coordinator and Adviser visits to Kano, Jigawa, Yobe, Katsina and Kano conducted during the reporting year
4. Written evidence from DFID health, nutrition, WASH and social development advisers, and regional coordinators
5. Field visits through out the year by the lead adviser to selected MNCh2 sites, primarily in Yobe, Kaduna and Kano, and meetings with government counterparts at state and federal level.
The final annual review report was completed by DFID.
Smart Guide i
Smart Guide The Annual Review is part of a continuous process of review and improvement throughout the programme cycle. At each formal review, the performance and ongoing relevance of the programme are assessed with decisions taken by the spending team as to whether the programme should continue, be reset or stopped. The Annual Review includes specific, time-bound recommendations for action, consistent with the key findings. These actions – which in the case of poor performance will include improvement measures – are elaborated in further detail in delivery plans. Teams should refer to the Smart Rules quality standards for annual reviews.
The Annual Review assesses and rates outputs using the following rating scale. ARIES and the separate
programme scoring calculation sheet will calculate the overall output score taking account of the weightings and individual outputs scores
Description Scale
Outputs substantially exceeded expectation A++
Outputs moderately exceeded expectation A+
Outputs met expectation A
Outputs moderately did not meet expectation B
Outputs substantially did not meet expectation C
Teams should refer to the considerations below as a guide to completing the annual review template.
Summary Sheet
Complete the summary sheet with highlights of progress, lessons learnt and action on previous recommendations
Introduction and Context
Briefly outline the programme, expected results and contribution to the overall Operational Plan and DFID’s international development objectives (including corporate results targets). Where the context supporting the intervention has changed from that outlined in the original programme documents explain what this will mean for UK support
B: Performance and conclusions
Annual Outcome Assessment
Brief assessment of whether we expect to achieve the outcome by the end of the programme
Overall Output Score and Description
Progress against the milestones and results achieved that were expected as at the time of this review.
Key lessons Any key lessons you and your partners have learned from this programme Have assumptions changed since design? Would you do differently if re-designing this programme?
How will you and your partners share the lessons learned more widely in your team, across DFID and externally
Key actions
Any further information on actions (not covered in Summary Sheet) including timelines for completion and team member responsible
Has the logframe been updated since the last review? What/if any are the key changes and what does this mean for the programme?
C: Detailed Output Scoring
Output
Set out the Output, Output Score
Smart Guide ii
Score
Enter a rating using the rating scale A++ to C.
Impact Weighting (%)
Enter the %age number which cannot be less than 10%.
The figure here should match the Impact Weight currently shown on the logframe (and which will need to be entered on ARIES as part of loading the Annual Review for approval).
Revised since last Annual Review (Y/N).
Risk Rating
Risk Rating: Low/Medium/High
Enter Low, Medium or High
The Risk Rating here should match the Risk currently shown on the logframe (and which will need to be entered on ARIES as part of loading the Annual Review for approval).
Where the Risk for this Output been revised since the last review (or since inception, if this is the first review) or if the review identifies that it needs revision explain why, referring to section B Risk Assessmen
Key points
Summary of response to iprogrammessues raised in previous annual reviews (where relevant)
Recommendations
Repeat above for each Output.
D Value for Money and Financial Performance Key cost drivers and performance Consider the specific costs and cost drivers identified in the Business Case Have there been changes from those identified in previous reviews or at programme approval. If so, why? VfM performance compared to the original aVfM proposition in the business case? Performance against vfm measures and any trigger points that were identified to track through the programme Assessment of whether the programme continues to represent value for money? Overall view on whether the programme is good value for money. If not, why, and what actions need to be taken? Quality of Financial Management Consider our best estimate of future costs against the current approved budget and forecasting profile Have narrative and financial reporting requirements been adhered to. Include details of last report Have auditing requirements been met. Include details of last report
E Risk
Output Risk Rating: L/M/H
Enter Low, Medium or High, taken from the overall Output risk score calculated in ARIES
Overview of Programme Risk
What are the changes to the overall risk environment/ context and why?
Review the key risks that affect the successful delivery of the expected results.
Are there any different or new mitigating actions that will be required to address these risks and whether the existing mitigating actions are directly addressing the identifiable risks?
Any additional checks and controls are required to ensure that UK funds are not lost, for example to fraud or corruption.
Outstanding actions from risk assessment Describe outstanding actions from Due Diligence/ Fiduciary Risk Assessment/ Programme risk matrix Describe follow up actions from departmental anti-corruption strategies to which Business Case assumptions and risk tolerances stand
F: Commercial Considerations
Delivery against planned timeframe. Y/N Compare actual progress against the approved timescales in the Business Case. If timescales are off track provide an explanation including what this means for the cost of the programme and any remedial action.
Performance of partnership How well are formal partnerships/ contracts working Are we learning and applying lessons from partner experience
Smart Guide iii
How could DFID be a more effective partner
Asset monitoring and control
Level of confidence in the management of programme assets, including information any monitoring or spot checks
G: Conditionality
Update on Partnership Principles and specific conditions.
For programmes for where it has been decided (when the programme was approved or at the last Annual Review) to use the PPs for management and monitoring, provide details on:
a. Were there any concerns about the four Partnership Principles over the past year, including on human rights?
b. If yes, what were they?
c. Did you notify the government of our concerns?
d. If Yes, what was the government response? Did it take remedial actions? If yes, explain how.
e. If No, was disbursement suspended during the review period? Date suspended (dd/mm/yyyy)
f. What were the consequences?
For all programmes, you should make a judgement on what role, if any, the Partnership Principles should play in the management and monitoring of the programme going forward. This applies even if when the BC was approved for this programme the PPs were not intended to play a role. Your decision may depend on the extent to which the delivery mechanism used by the programme works with the partner government and uses their systems.
H: Monitoring and Evaluation
Evidence and evaluation Changes in evidence and implications for the programme Where an evaluation is planned what progress has been made How is the Theory of Change and the assumptions used in the programme design working out in practice in this programme? Are modifications to the programme design required? Is there any new evidence available which challenges the programme design or rationale? How does the evidence from the implementation of this programme contribute to the wider evidence base? How is evidence disaggregated by sex and age, and by other variables?
Where an evaluation is planned set out what progress has been made.
Monitoring process throughout the review period. Direct feedback you have had from stakeholders, including beneficiaries Monitoring activities throughout review period (field visits, reviews, engagement etc) The Annual Review process
