The Provincial-wide Monitoring and Evaluation System ... · The planned system is to enable a...

TTHHEE PPRROOVVIINNCCIIAALL--WWIIDDEE MMOONNIITTOORRIINNGG AANNDD EEVVAALLUUAATTIIOONN SSYYSSTTEEMM

BBUUSSIINNEESSSS RREEQQUUIIRREEMMEENNTT

SSPPEECCIIFFIICCAATTIIOONN

[BRS]

Department of the Premier in collaboration with University of the Cape Town, Southern Africa

Labour and Development Research Unit [SALDRU]

Doc Name:Doc Name:Doc Name:Doc Name: M& E business requirement spec of 33 Last Updated: Last Updated: Last Updated: Last Updated: on 19/04/2010 at

MMMMMMMMOOOOOOOONNNNNNNNIIIIIIIITTTTTTTTOOOOOOOORRRRRRRRIIIIIIIINNNNNNNNGGGGGGGG AAAAAAAANNNNNNNNDDDDDDDD EEEEEEEEVVVVVVVVAAAAAAAALLLLLLLLUUUUUUUUAAAAAAAATTTTTTTTIIIIIIIIOOOOOOOONNNNNNNN BBBBBBBBUUUUUUUUSSSSSSSSIIIIIIIINNNNNNNNEEEEEEEESSSSSSSSSSSSSSSS RRRRRRRREEEEEEEEQQQQQQQQUUUUUUUUIIIIIIIIRRRRRRRREEEEEEEEMMMMMMMMEEEEEEEENNNNNNNNTTTTTTTT SSSSSSSSPPPPPPPPEEEEEEEECCCCCCCCIIIIIIIIFFFFFFFFIIIIIIIICCCCCCCCAAAAAAAATTTTTTTTIIIIIIIIOOOOOOOONNNNNNNN

Table of Contents

1 Executive Summary............................................................................................. 4

2 Introduction/background ..................................................................................... 5 2.1 Purpose of this document..................................................................................5 2.2 Background ...................................................................................................5 2.3 Information Sources Used..................................................................................6

3 Scope .............................................................................................................. 7 3.1 Scope – Context of Request................................................................................7 3.2 Exclusions.....................................................................................................7 3.3 Information collection......................................................................................7 3.4 Assumptions and Dependencies ...........................................................................7 3.5 High Level Requirements (What is In) ...................................................................7 3.6 Excluded Requirements (What is Out) ...................................................................8 3.7 Design Constraints...........................................................................................8 3.8 Critical Success Factors ....................................................................................9 3.9 Solutions Considered and Discarded .....................................................................9

3.9.1 Solution 9 4 Details of Requirements ......................................................................................11

4.1 Business Value Chain...................................................................................... 14 4.1.1 Context Diagramme ......................................................................................... 17 4.1.2 Process Model ............................................................................................... 18 4.1.3 Business Requirements..................................................................................... 20 4.1.4 Business Specification ...................................................................................... 22

4.2 Business Information Requirements .................................................................... 24 4.2.1 Key Performance Indicators................................................................................ 24 4.2.2 Main Data Entities ........................................................................................... 24 4.2.3 Sources of Data.............................................................................................. 25 4.2.4 Data Quality management: ................................................................................. 27

5 Implementation Considerations ............................................................................28 5.1 Transformation Management............................................................................ 28 5.2 Business Procedures....................................................................................... 28 5.3 Training ..................................................................................................... 29 5.4 Quality Assurance ......................................................................................... 29

6 Appendix 1: Revision History ...............................................................................30

7 Appendix 2: Approval to Proceed/Sign-off Approval not to proceed ..............................30

8 Appendix 3: Distribution .....................................................................................30

9 Appendix 4: Acronyms .......................................................................................31

10 Appendix 5: References ......................................................................................32

Deleted: 14/04/2010



Table of Figures

Figure 1: M and E.............................................................................................................. 8 Figure 2: Diagramme: M and E Future System Model ..................................................................11 Figure 3: Information flow across directorates .........................................................................12 Figure 4: Monitoring and Evaluation Value Chain.......................................................................14 Figure 5: Process and Information Flow Diagramme...................................................................15 Figure 6: Context Diagramme..............................................................................................17 Figure 7: High level process ................................................................................................18 Figure 8: Reporting mechanism process..................................................................................19 Figure 9: MER Context Diagramme........................................................................................21 Figure 10: Indicator Development Process...............................................................................22

Deleted: 14/04/2010



1 EXECUTIVE SUMMARY

Information Technology plays an essential role in the growth and survival of organisations today. It is through collaboration with business that IT creates value in the form of improved business processes, new products and services in response to changes in the business environment. Since data and information are the lifeblood of virtually all business practices, ICT projects are often the key mechanism used to turn an organisation's vision and strategy into reality. An electronic, web enabled Provincial-wide Monitoring and Evaluation (M and E) system (PWMES) was requested to effectively assist with managing, collecting data and reporting on the Monitoring and Evaluation activities. The planned system is to enable a seamless flow of information and data between various sources, i.e. 12 Provincial Departments together with various stakeholders. This system will then generate reliable and relevant data and reports as and when needed. The main challenge is that the Monitoring and Evaluation within Provincial Government Western Cape (PGWC) is not taking place in the context of the Provincial-wide Monitoring and Evaluation System. The present monitoring and evaluation processes and data source systems lack integration and coherence. It is against this view that the PGWC has decided to embark on a project of this magnitude. This document will provide the detailed technological requirements to the processes of the Provincial-wide Monitoring and Evaluation System. The requirements would ensure that the process system is translated into an automated system.

Deleted: 14/04/2010



2 INTRODUCTION/BACKGROUND

2.1 PURPOSE OF THIS DOCUMENT

The purpose of this document is to provide detailed business requirements for the Provincial Wide Monitoring and Evaluation system.

Currently the PGWC has been mandated to have a Monitoring, Evaluation (M&E), Review and Reporting function. The main objectives of this Chief Directorate (CD) are to monitor, evaluate, review and report on the strategic goals and objectives1 of the 5 year Provincial Strategic Plan using indicators as the key instrument of measurement.

The Provincial-wide M and E System for the Western Cape has adopted a results-based approach which monitors, evaluates, review and report on the outcomes and impacts of the strategic goals and objectives of the Provincial Strategic Plan via key strategic priorities. On implementation level the Provincial-wide M and E System, reports against the progress of key programmes and projects as specified in the strategic goals and objectives of the Provincial Strategic Plan. For this endeavor to be successful it is imperative that it has the necessary technological support. The CD: Monitoring, Evaluation and Review has partnered with the Branch: e-Innovation in order to deliver an integrated and coherent system that will support the processes of collecting, measuring, monitoring, evaluating, reviewing and reporting of data and information. There exists a need in the Provincial Government of the Western Cape to improve and enhance systems and practices of M and E that are essential for long-term strategic planning and development. The Provincial-wide M and E System should be closely structured around various components namely the Provincial-wide M and E Strategy Frameworks, the M and E elements, mechanisms and tools and the reporting cycle. These components should operate within the context of the existing governmental planning framework cycle of events. The PWMES should be clearly integrated within the government planning cycle and complement it wherever possible.

This document provides the details of the following: - The nature of the system - The functionalities of the system - The scope of the system - The data and informational needs - The internal and external stakeholders

2.2 BACKGROUND

The findings of a rapid review of existing South African government M and E systems, undertaken by the National M and E task team, indicates that M and E Systems are generally underdeveloped and inadequate, although basic elements are usually present in line with governments strategic planning and budgeting systems. [Source: The Presidency of the Republic of South Africa (2005) – Proposal and Implementation Plan for a Government–wide Monitoring and

Evaluation System. The Presidency Policy Unit, Pretoria] The Document entitled, Current Reality of Monitoring and Evaluation in the Western Cape, reflects on the will and capacity of provincial government departments to monitor and evaluate their own

1 The strategic goals and objectives refer to the international, national and provincial plans and targets of

the PGWC Deleted: 14/04/2010



performance. It presents a picture of an organization which understands and supports the need for monitoring and evaluation processes, and which is well supported in some aspects of M and E performance measurement. It also demonstrates a lack of a coherent framework to ensure that monitoring and evaluation activities contribute to strengthening holistic governance. A clear request for the Department of the Premier to provide leadership and co-ordination was articulated during the reported consultative process. [Source: Department of the Premier: CD: MER:

Current reality of M and E in the Western Cape province. 2005 p3] The findings of the Provincial Audit Phase One served as a detailed scoping exercise for establishing a comprehensive M and E study across all M and E units in the province. The Provincial Audit Phase Two identified a number of critical challenges facing the establishment of a PWMES, detailing capacity constraints, strategy alignment problems and reporting inconsistencies. . [Sources: Department of the Premier: CD: MER: Current reality of M

and E in the Western Cape province. 2005 p3 and Department of the Premier: CD:MER:

Provincial Audit Phase 2, 2007 p11-12]

2.3 INFORMATION SOURCES USED

In order to develop the business requirements for the PWMES, background documentation regarding the proposed system was sourced from the CD: Monitoring, Evaluation, Review and Reporting.2 In addition, a series of workshops were held over the period 2006 to 2008 to understand the work processes of the Chief Directorate. A series of stakeholder engagements occurred across provinces including a site visit to Malaysia to understand the Malaysian M and E system model. During the period 2007-2008, the discussions and workshops focused on developing business process maps for what is required to conduct integrated and coherent monitoring and evaluation. These process maps included all the functions of the directorates which are involved in the entire M and E end-to-end business process. The results-based monitoring and evaluation approach to the M and E requirements of government’s policies, programmes and projects serves to support decision-making, to improve accountability and continuous learning. On a transversal level Monitoring and Evaluation should affect all business units of the 12 Provincial Government Departments. Furthermore, these provincial government departments may assign the M and E aspect to different high level strategic committees to execute and chart the way for effective M and E. Currently the CD: Monitoring, Evaluation, Review and Reporting has established the following internal and external stakeholders

- Provincial M and E Forum - 12 Provincial Departments – including Provincial Parliament - The Presidency - Statistics SA - National Departments - Non Governmental Institutions - Universities

2 The information sources include all the reference material of the Provincial-wide Monitoring and

Evaluation System Deleted: 14/04/2010



3 SCOPE

3.1 SCOPE – CONTEXT OF REQUEST

The overall system objectives should be the collection, collation and analysis of the data and information for the core set of indicators developed for the strategic goals of the PSP. The application of the data and information relates to the progress of programmes and initiatives in order to:

- Ensure transparency and accountability - Promote service delivery improvement - Ensure compliance with legislative requirements - Assist in the implementation of the strategic objectives of the Provincial Strategic Plan.

3.2 EXCLUSIONS

It should be noted that collection of information will be excluded from this project – this function will be performed by departments. The departments will then feed their information to the overall Monitoring and Evaluation System.

3.3 INFORMATION COLLECTION

The System will draw information from a range of sources and present this in user-friendly and accessible formats tailored to the needs of different users such as the Premier, the Presidency, the Director-General, etc. It is proposed that the system should make use of a web-based system onto which institutions i.e. departments will be expected to post or submit their information or reports. Where a strong M and E function exists, there may be a possibility of direct integration with that particular system or function.

3.4 ASSUMPTIONS AND DEPENDENCIES

In order for this system to be fully functional, the following factors or assumptions are made: - Integrated approach in the developing and review of developmental 5 year Provincial

Strategic Plan - Departments strategies are linked to the strategic objectives of the Provincial Strategic

Plan - A compendium of indicators are developed, aligned and updated - A fully functional M and E function exist within different departments - A proposed system design will be signed off - Clearly defined reports and reporting structures will be created

3.5 HIGH LEVEL REQUIREMENTS (WHAT IS IN)

A system, which will contribute to improved governance and enhance the effectiveness of the provincial government and its institutions, is required.

Deleted: 14/04/2010



The system should make it easier for decision-makers to have a snapshot view of the progress and problem areas as per the approved Provincial Strategic Plan. The system should support the business model as shown below:

Cluster level

Departments

Line Departments

Resul ts bas ed M a nd E:

Proje cts

Policy De velopment 5 ye ar

Prov Strategic PlanPolicy

Implement at ion

Support

Monitoring,

Evalua tion and

Revi ew

Communication

DGPTM

M and E of pro jects

Out put a nd ou tcome

indic ato rs

Impl ementa tio n M a nd E

Political Stakeholders

Figure 1: M and E Process

The PWMES must allow for the following:

- an information flow between different spheres of Government - Assist with assessment and accreditation service so that it is clear when the necessary

standards are met and whether improvements are required - Allow access to clear, accurate and well presented updates on governmental projects.

The PWMES can facilitate the functioning of the executive projects dashboard across departments, within and across projects where necessary. This responsibility falls broadly within the Directorate Policy Implementation and Support. This facilitation could be communicated to all relevant stakeholders, but essentially it should feed into the CD: MER, which has a ‘hypothetical’ transversal link across directorates under the DG, its associated functions and the provincial line departments in the province.

3.6 EXCLUDED REQUIREMENTS (WHAT IS OUT)

The proposed system will be a secondary system (within a provincial context) in that it does not undertake primary M and E functions within Departments. The exclusions are:

- Research and data collection - Verification of information - Developing, and maintaining an effective indicator system - Developing and constructing databases

3.7 DESIGN CONSTRAINTS

System integration remains the huge challenge in making sure that there is a seamless information flow. Currently there are numerous M and E stand alone systems which need to be

Deleted: 14/04/2010



linked and aligned to the newly designed Monitoring and Evaluation system. Limitations on the capacity from a technological and infrastructural perspective may hamper the progress of this system design.

3.8 CRITICAL SUCCESS FACTORS

- Fully functional departmental M and E systems - Integrated strategic planning between departments - Adherence to the governmental cycle of event/timeline - Common base for reporting and timelines - Well developed compendium of indicators (Departmental; Government-wide;

Provincial –wide; Provincial Programme of Action) - Accurate and reliable information

3.9 SOLUTIONS CONSIDERED AND DISCARDED

3.9.1 SOLUTION

The proposed system is based on promoting and managing certain information collected from various sources. The system will assist in analysing and generating reports that address overall government M and E performance, institutional performance; progress in implementing programmes and measures; and the outcomes and impacts of the Provincial Strategic Plan. It will also receive impact evaluation reports and make these available to system users. The solution is trying to address these issues:

Issue Current State Future State Beyond Monitoring

Most systems are reporting systems focused on “Monitoring” of indicators.

Systems support for 1. Monitoring, 2. Evaluation and 3. Review and Reporting.

Silos M and E processes and systems within the departments are designed and implemented as "silos". Databases and systems are not integrated.

Transversal processes and systems to integrate and align M and E across the province.

Compliance Based

Reporting is driven by compliance to indicators rather than outcomes.

Implementation of a Results Based Monitoring and Evaluation (RBM and E) model. Collaborative and knowledge based system and process to keep the focus on the objectives, desired outcomes and identifying impact.

Common Reference

There is a lack of common definition, method and reference for cascading and mapping objectives, identifying desired outcomes, impacts, targets and indicators.

A transversal framework of objectives, outcomes, impact, targets and indicators. The Provincial-wide M and E Forum and Seminar Series aims at identifying overarching transversal objectives, the compendium of indicators and associated information.

Information There are known and common Standardisation of information Deleted: 14/04/2010



Flow statistical information to support indicators. However, there is a need to integrate and automate the information flow which is duplicated, becomes static and costly to maintain.

interchange and interfaces, workflow and process automation to enable process and information flow.

Knowledge System

Systems do not support collaborative, research and learning aspects of M and E

Web based platform with content management, search, repository and collaboration tools.

Tracking and Auditing

Changes are difficult to manage or track.

Processes to support tracking and auditing of change.

User Platform Disparate systems and user platforms.

A common user platform using Internet based technologies and portals such as Internet, Extranet and Intranet. Using browser based online services as tools to implement a knowledge based M and E system.

Process Automation

Tedious capturing and reporting processes.

Automation using common and re-usable processes based on Internet technologies such as XML web services.

Online Services and Tools

Lack of tools to support workflow, collaboration, schedules and research.

Rich internet applications to support scheduling, documentation, reporting, information search, workflow and collaboration.

Stakeholder Interaction

No mechanism to involve stakeholders such as the Citizen or the “Customer”.

Integrated stakeholder interaction with online forums, communication and media.

Integration into core functions

M and E is not integrated into other core functions

M and E processes integrated into other processes and systems such as outcomes based programme management.

Over Reporting Burden of governance and reporting on limited resources which need to focus on service delivery.

Systems and automation to lessen rather than increase workload.

Deleted: 14/04/2010



4 DETAILS OF REQUIREMENTS

Data Interchange

Source Systems: Departmental M & E systems

M and E Information SystemsM and E

Repository

External interfaces: National; Stats SA, etc.

Figure 2: Diagramme: M and E Future System Model

The above Diagramme (figure 2) shows the M and E information system front end in blue. The Source systems are all 12 Departments including the Local Government sphere feeding information to the M&E System. Since the systems used by the Source Systems might be different, there is a need for a Data interchange to convert data from source into the M and E information system.

External interfaces such as National Government Departments, Statistics South Africa and other agencies will also be feeding information to the system. The processes and procedures for this system should be based on the government calendar of events as depicted below. The diagramme is showing the combination of Human Capital, processes and Technology and their level of involvement in executing different tasks associated with Monitoring and Evaluation cycles.

It is anticipated that the PWMES will be web-based and act as a portal for sending and receiving information. The intention is not to warehouse any source of data, including provincial departments. The PWMES plans to achieve an exchange of data/and information given the need and its application. For example, the Department of Health will always be the custodian of infant and maternal mortality data but it may be required by another party, a political principal to update

Deleted: 14/04/2010



the MDGs for instance. The idea is not to duplicate data unnecessarily as warehousing would imply maintenance and storage costs including related skills to do the necessary.

Government Planning

Cycle

Review and Reporting Cycle

Evaluation Cycle

Monitoring Cycle

Calendar of Events

Web Based System RBM and E Portal:

Portal and Online Services to support Knowledge Workers.

Tool with Content and Document Management Capability.

Event Driven Transactions and Workflow for Results Based

Monitoring and Evaluation within Government Planning Cycle.

View Event

Attach Documents

Compose Report

Search Compendium

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Five Year Electoral Cycle | Five Year Medium-Term Strategic Framework (MTSF)

Three Year Medium-Term Expenditure Framework (MTEF)

Annual Budgetary Cycle | Annual Parliament Cycle

Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

National Government

Provincial Government

Local GovernmentAcross 3 Spheres of Government

Human Organisation Process and Information Documents and Work ProductsInformation Systems

Figure 3: Information flow across directorates

In the context of this Business Requirement Specification and within the CD: MER the above illustration (figure 3) at face value demonstrates the information flow across directorates within this Chief Directorate as informed by the other line departments of this province. The four common denominators across monitoring and evaluation are:

� Human organization, � Process and information, � Information systems, and � Documents and work products.

This could also be said of any other line department as well to differing degrees. Despite this illustration being one dimensional in its presentation, it could be seen in a cyclical manner against a particular timeframe, i.e. the calendar of events according to the government planning cycle. The driver as it were of this ‘cycle’ is the web based system RBM & E portal with its supporting elements [on line service to support knowledge workers, content and document management capability tool, event driven transactions and workflow for results based, monitoring and evaluation with Government planning cycles] as indicated.

Deleted: 14/04/2010



PWMES: Development, Protocol and Procedure

CD: M and E as a System owner 12 Provincial departments

Communicate with all affected stakeholders and ensure they

are fully aware of what is required of them

Ensure compliance to the Guidelines

for reporting from norms and

standards as indicated in the

Provincial-wide Monitoring and

Evaluation Framework

Providing the information technology infrastructure for the

submission of information and the creation of system reports

Provide the necessary information in

agreed formats and standard as

required

Inform the relevant authorities and executives when system

information indicates a problem requiring attention (strategic

issues)

Respond appropriately to information

received

Developing and improving the system overtime

Deleted: 14/04/2010



4.1 BUSINESS VALUE CHAIN

Monitoring and Evaluation business value chain is presenting the context in which the business unit is operating. This is presented by a mixture of Core Processes and Support Processes to provide a desirable output.

Monitoring and Evaluation

Dev elop and implement Prov incial Wide policies, strategies and programs f or Monitoring and Ev aluationEnsure compliance to the National Gov ernmental Monitoring, Review and Reporting standardsEnsure continuous monitoring, ev aluation and review of the Prov incial Strategies

Support Processes

Determine information requirements for the purposes of monitoring within the MERR.Direct the management of the provincial monitoring sys tem in line with strategic goals and objectivesEnsure provincial alignment with the nationally prescribed M and E framework

Develop compendium of indicators and core directory of common

data sources

Indicators M and E indicator system for each core indicatorMonitoring and Results Framewor kCore director y of common data sources for iKapa GDS

Provincial MonitoringProvincial Evaluation

Determine evaluation requirements for purpose of MERREnsure i ntegrated evaluation processes in line with the s trategic goals and objecti vesEnsure provincial alignment with the nationally prescribed M and E framework

Provincial Review and Reporting

Information on MDGs, GDS and 5 Year Provincial Strategic Plan as well as governance (indicators)Inform implementation of Provincial strategiesImprove transparencyPromote service delivery improvement

Outputs / Outcomes

Infor mati on Architecture to support the PWMESDepartmental SystemsExternal Systems

Inter nal M and E forum representing provincial governmentNetwor k of monitoring and evaluation practitioners and special interest groups

To facilitate the implementati on of monitoring and evaluation methodologies to achieve RBM and EFacilitate wor kshops on monitoring and evaluation

Provide an effecti ve communication ser vice and collaboration with stakeholders r egarding M, E and R matters

Strategic Leadership and Support on M and E processes

Provincial wide Monitoring and

evaluation system

Monitoring and Evaluation Forum

Capacity Building CommunicationOther Provincial Strategies

Alignment of a monitoring, evaluation and reporti ng function with other provi ncial secondar y strategies

Core Processes

Ensure provincial alignment to the GWMES framewor kDevelop provincial MERR framewor ks, systems and proceduresProvide leadership to M and E organisational structures

Determine reporting requirements for purpose of MERREnsure i ntegrated evaluation processes in line with the overarching goals and objec tivesEnsure provincial alignment with the nationally prescribed M and E framework

Figure 4: Monitoring and Evaluation Value Chain

It is evident from the above illustration for Monitoring and Evaluation in a RBM & E environment where Monitoring, Evaluation, Review and Reporting is done on a continuous basis that core processes are essential to achieve departmental outcomes such as improving transparency, to promote service delivery in this regard for instance:

- to develop a compendium of indicators and a core directory of common data sources which is governed by the Monitoring and Results frameworks

- to ensure provincial alignment to the Government-wide Monitoring and Evaluation System (GWMES) framework with a view to develop M and E and Review and Reporting frameworks, systems and procedures at provincial level. In addition, to provide the necessary leadership to M and E organizational structures

Deleted: 14/04/2010



- alignment at and to national level will be underpinned by integrated Monitoring, Evaluation, Review and Reporting processes in line with the 5 year Provincial Strategic Plan which will in turn determine M and E requirements for MERR at provincial government level.

Information architecture is seen as a support process in that it would strategically link the PWMES to all departmental systems and external organisations/stakeholders elsewhere. Engagement and interaction will not only occur at an electronic/digital level but would be enhanced by a Provincial M & E forum representing a provincial government network of monitoring and evaluation practitioners and special interest groups. These processes as outlined above (figure 4), will be accompanied by capacity building exercises via appropriate workshops through an effective communication platform with relevant stakeholders in this regard.

This Business Model will increase the Government’s effectiveness, so that it can achieve its desired outcomes and the strategic goals of the 5 year Provincial Strategic Plan. The system then uses the Strategic framework as a starting point in determining the monitoring and evaluation criteria. The diagramme below shows how the linkages and flow of information will occur.

Figure 5: Process and Information Flow Diagramme

This illustration above (figure 5) could be viewed as a dynamic process flow in an arterial manner as it were. The arteries are the ‘sources’ outside the ‘rectangle’ shaped block which contribute to

MERR ,

Processes

Department S t a t i s t i c a l

I n f o ( d a t a

m a r t s )

T e m p l a t e

C o m p l e t e d T e m p l a t e

P r o j e c t I n f o &

I s s u e s

D e p a r t m e n t a l

P e r f o r m a n c e i n f o

R e q u e s t i n f o

r m a t i o n f o r

e g . B u d

g e t

S p e e c h / s t a

t e o f t h e

P r o v i n c e A d d

r e s s

P e r f o r m a n c e

I n f o

DG/Premier

Citizen Issue

System

Permis

Data Source

Cluster Dashboard System

Deleted: 14/04/2010



the data and information gathering processes. These sources could be independent of the MERR process, a process in its own right except the MERR process wants to align and work towards seamless integration of common and disparate data sources for that matter where needs be.

Deleted: 14/04/2010



4.1.1 CONTEXT DIAGRAMME

P W M ES

F r am ewo r ks

Co nce ptua l F ra mew or k

M ER F ra me wor k

M ER S tra tegy

Me ch a ni sm s

Dep ar tmen ta l M a nd E S ys tem s

Data A sse ssme n t P ro cess

M an d E P e rfor ma nce No rms and S ta nd ar ds

Cap acity bu ild in g

M an d E F o ru m

M AN D E E L E ME NT S

In d icat o r dev el op me n t

GDS /M DG’s

Le ad in ter ve ntion s

SOPA indi cato rs

P ro v 5- ye ar S tr ate gic A ge nd a

Mo n ito r in g an d Resu lt s F ra me wo rk

Dat a Man a ge m ent

Ma instr ea m

P ro vincia l Ad min istra tive

GDS Bas elin e s

Mo n ito r in g

E val ua t io n

Rep o rt in g

I nf or m at io n Arc h it e ctu r e

Figure 6: Context Diagramme

The above diagramme (figure 6) not only captures the context within which the PWMES is operating but the building blocks of each component to ensure integrated and coherent RBM + E as indicated below:

- Under the broad category M and E Elements: the Indicator Development and Monitoring and Results frameworks, Provincial 5-year Strategic Agenda and SOPA indicators are vital. Data management for mainstream and provincial administrative data level is a requirement.

- The Information Architecture to be supported by a core directory and a compendium of indicators

- Frameworks and strategies for PWMES to be in place - Mechanisms to ensure systems, norms, standards to enhance capacity building,

Deleted: 14/04/2010



4.1.2 PROCESS MODEL High level process for CD: MER

Figure 7: High level process

The high level process, as depicted above (figure 7) for the CD: Monitoring, Evaluation, Review and Reporting, is pivotally informed by the MDGs, GDS and PGWC strategic goals and

objectives, its objectives contained therein which is aligned to the Provincial Programme of Action with its supporting PGWC strategic goals and objectives. Given the fact that a results based approach has been adopted with reference to monitoring and evaluation, indicators are crucial in monitoring the progress, outcomes and outputs as it were of the MDGs, GDS and PGWC strategic goals and objectives. With reference to indicators in particular, a compendium of indicators were developed linked to its primary sources and a resultant database was designed to structure and streamline the indicators. In terms of the M and E tools and methodologies which sustain and informs the effective indicator system, it also play a crucial role in the design of ‘pre-populated’ instruments or templates that makes it possible for results-based monitoring and evaluation to occur, including data and information verification. Evaluation assessment can be done at implementation and results-based level respectively which ultimately leads to review and assessment feedback that is fed to the PGWC strategic goals and objectives.

Indicators

Compendium of Indicators

Reference Sources

Goals

Objectives

• Pre-populate instrument

• Request Departments to

completeFor completion

M, E & R Tools and methodologies

PPOA

-Lead Intervention

- Priority projects

• Process data and information received

• Verification of data and information

1. Evaluation assessment of the data & information; - Implementation level

- Result based levels

Develop, maintain and sustain an effective indicato r system

Databases

Review and assessment feedback

Deleted: 14/04/2010



Figure 8: Reporting mechanism process

The Diagramme above (figure 8) illustrates the reporting mechanism process with the Provincial Departments. As for this step of the reporting process, an Excel spreadsheet/template is designed with the progress indicators in mind in collaboration with provincial departments concerned.

Step 1: In general this step is intended to set the tone as it were to the department concerned and allows MERR directorates to have a basis to work from. In terms of programme implementation/transversal programmes these tools are pre-populated with progress indicators in hand. Step 2: After having pre-populated a template/an instrument for a particular programme, it is forwarded to the department [s] concerned to be completed. Step 3: The ‘responses’/input received from the department[s] concerned are processed and consolidated. Step 4: Verification is done as data/information has been processed to a point. Step 5:

Pre-populate Tools with

Progress indicators

1

Forward to

depts. & consult for completion

2

Process & consolidate

data

3

Verify data & information

4

Compile reporting

assessments

5

Compile evaluation

assessment

6

Forward reports to

stakeholders

7

Deleted: 14/04/2010



Reporting assessments are compiled which in turn are forwarded to the evaluation directorate. Step 6: Evaluation does an assessment which could be process, compliance or implementation driven given the requirement in hand. Evaluation then forwards the verified, assessed template to Review and Reporting directorate to be amended and/or to incorporate the changed template as received from departments concerned and having undergone scrutiny by the other directorates. Step 7: The processed/compiled report after having been validated gets sent to the stakeholders concerned.

4.1.3 BUSINESS REQUIREMENTS

1. Transversal systems must be integrated and align M and E across the province. These systems are:

- Treasury: Value for money - DPSA: Human resources utilisation and Early Warning (with Treasury) - PSC: Public Administration barometer - Department of Justice: Constitutional Rights - DPLG: Service delivery quality - Stats SA: National Statistics System - GCIS: Government Information system support

Deleted: 14/04/2010



Figure 9: MER Context Diagramme

2. The system must provide support for:

� Monitoring � Evaluation � Review and Reporting.

The MER Context Diagramme, (Figure 9) from left to right, could be viewed as the external environment, ranging from external systems [multiple and varied], departmental M and E systems, QPRs and the Citizen Engagement process. In brief, these are the primary contributors as it were to the internal environment of the MER context diagramme, including the core directory.

The directory is a synthesis of the primary contributors’ inputs which provides a platform for results based monitoring and evaluation/RBM and E, namely measurable indicators.

The right-hand side is essentially the key stakeholders to whom data and information is provided from the MERR functions as indicated within the ‘core structure/battery shape’ image here. These stakeholders including those that are external, like universities, Stats SA and other independent organizations/institutions provide information to provincial departments as well as the political principals in the province (i.e. the Director-General, the Premier, etc.)

Essentially the external stakeholders would be interested in feedback on their products as well as new sources of data that could act as a cross-reference to data/information or mainstream data in some instances.

Deleted: 14/04/2010



Feedback is obtained from provincial departments via progress report cards whereas reports could be used for departments in return for developmental indicators, [MDGs for example] and functional indicator reporting instruments, like a particular indicator in the agricultural sector for instance.

Figure 10: Indicator Development Process

The system must provide for the provincial wide co-ordination and facilitation as per the top segment of the Diagramme above (Figure 10) which will be reinforced by networking and collaboration with internal and external stakeholders to ensure an effective implementation of the provincial wide M and E Strategy. The Provincial M and E Forum is a platform for internal stakeholders per se within provincial government by means of a partnership between the Department of the Premier: CD: MER and the Departmental M and E staff. With reference to an effective indicator system, the system must assist with the development and refinement of output and outcome indicators. The review of indicators needs to be continuous and it must be maintained to be sustainable and relevant.

The system must collate, interpret and calibrate the selected data into output and outcome indicators as per the bottom segment of the Diagramme (Figure 8) above, with reference to MERR on Implementation and Results-based level. These indicators inform and feed as it were the PWMES which are RBM and E-oriented. Monitoring- extracts from the core directory list of ‘select’ developmental (results-based level) and functional (implementation level) data is informed by external [mainstream] and internal [administrative] data sources for that matter.

The system must provide the ability to perform evaluation. Where relevant M and E data is utilised which could have been provided by provincial departments concerned directly, or by monitoring and reporting directorates, specific data collected for evaluation uses the most appropriate and relevant methodology and technique. Periodic, annual and long term assessments are conducted by the Directorate Evaluation.

The system must provide the ability to create reports – generally reporting relies on the Annual Performance Plans and quarterly reports from departments to pre-populate a template in order to conduct periodic, annual and long term assessments [in conjunction with Monitoring and Evaluation directorates]. The review process feeds the recommended learning to support the revisiting of the policy and programme planning.

4.1.4 BUSINESS SPECIFICATION

4.1.4.1 FREQUENCY OF USE

This system will be updated and used on a daily basis to support management decisions, by recording any action or activities happening in monitoring and evaluation.

4.1.4.2 ACCESS SECURITY

Users will be granted passwords and access to various levels (which will limit what they can see and modify) according to their job duties. The users will be mostly:

Provincial Wide

Co-ordination and Facilitatio

Effective Indicator System (Development

& Functional)

Monitoring

Evaluatio

Reportin

M&E Forum: Partnership role between Dot

Departmenta

M&E

M o n i t o r i n g & E v al u at i o n ( M & E )

Network & Collaboration

Internal & external Stakeholders

Ensure

Departmental M&E Framework is in

Plac

Ensure that

Mechanism is

Plac

Ensure that

Elements is in Place

Develop Indicators

Review Indicato

r

Develop Priority Projects

Update & Align Indicators (APP; PSP

Capture Set Core of Indicators Cor

Director

Select Development & Functional Data Collect Data Interpre

Data Calibrate Indicators

Use relevant M&E Data

Conduct Periodic Assessments

Conduct Annual Assessments

Conduct Long Term Assessments

Pre-populate templat

Conduct Periodic Assessments

Conduct Annual Assessments

Conduct Long Term Assessments

Indicator

Results Framework

Identify & Locate/collect Data Sources

APP

Deleted: 14/04/2010



- Monitoring and evaluation personnel - Project managers – Departmental Monitoring and evaluation - Directors - Chief Information Office - Head of Departments

4.1.4.3 INPUT AND VALIDATIONS Validation of data according to the required strategic objectives monitored or evaluated. The monitoring indicators will also be useful in validating information.

This section will be provided in details on the System requirement

• Input screens • Codes and Descriptions • Field Validations • Mandatory Fields

4.1.4.4 OUTPUT

A comprehensive reports and information that will assist decision makers, project managers, legislators to make informed decision.

� Output report

The system should provide various reports such as

• Management Reports • Quarterly monitoring reports • National Programme of Action • Evaluation reports • Other

-

� Output data

The system will carry quarterly monitoring data, evaluation data and outcomes data. The data for these reports will be from all affected areas (Departments) as per the Strategic objectives.

� Output state

The system will provide its users with data for use in evidence-based decision-making in their own areas of responsibility. This will lead to improved decision making, better term use of resources and an increased focus on institutions requiring attention and support.

4.1.4.5 BUSINESS PROCESSING

The purpose of the PWMES is to deliver useful information and analysis to improve the monitoring and evaluation practices and capacity, contribution to a better public management. Effective public administration involves planning that leads to implementation and evaluation. Implementation of strategic objectives should be monitored and evaluated, producing knowledge and insights that are fed back into planning process.

Deleted: 14/04/2010



4.1.4.6 BUSINESS RULES

• Integrated Strategic Planning • Clear information requirements from departments and other

stakeholders • All monitoring and evaluation processes informed by planning initiatives

in place • The system explicitly promoting public participation in governance • Clear access to accurate and well presented updates on progress in

government programmes and their impact.

4.1.4.7 RESULT

Accurate and reliable information on progress in the implementation of government and other public sector programmes, collected and updated on an ongoing basis

4.1.4.8 REQUIRED ACTION

The development of clear M and E requirements and standards to be met by all provincial institutions, linked to the requirements of the National Indicator Initiative.

I 4.2 BUSINESS INFORMATION REQUIREMENTS

4.2.1 KEY PERFORMANCE INDICATORS

- Extent of compliance to the National M and E requirements by provincial entities - Frequency and quality of reports produced by provincial entities and transversal M and E systems - Quality and completeness of reports - Policy and strategy revision resulting from system reports

4.2.2 MAIN DATA ENTITIES

The main data entities for the PWMES are captured into the Core Directory. It must be kept in mind that the Core Directory is structured according to themes broadly, ranging from:

• employment and economic activity • education and skills development • unemployment and non-economic activities • poverty • health • transport and travel • basic services • crime • youth • quality of life • population and migration

Deleted: 14/04/2010



Another component to the Directory is provincial administrative data, which is work in progress and being profiled in stages. The former category is primarily mainstream data that is readily available whereas provincial administrative data resides mainly within the domain of provincial departments. One would find overlap in terms of data sources across the various themes as these sources/surveys cover different modules as it were, from employment and educational background to household size.

The Core Directory indicates to the reader the limitations of data sets/sources in hand, its application and at times incompatibility for that matter.

4.2.3 SOURCES OF DATA

The limitation within data sources is that they are not always useful and appropriate to use on a geographical level lower than provincial level. The following sources existing within the country are potential sources of good data that could be useful for the province as baselines for PWMES:

Administrative data

o Vital statistics, birth and death registration o Health Management Information Systems o Department of Health Information Systems (DHIS) o Ante Natal Care (ANC) o Nutrition Survey o Provincial Departments Annual report o School and Tertiary institutions data (CEMIS/FETMIS)

Financial Data

o Business, Retail and Industrial Surveys o Investment Climate Surveys o Macro data such as National Budgetary data, balance of payments, etc o Other such as micro-enterprise data

Household data

o Quarterly Labour Force Survey (QLFS) o General Household Survey (GHS) o Survey of Employed and Self Employed o SA Demographic and Health Surveys (SADHS) o Nelson Mandela Study Medical Research Council (MRC) on HIV/AIDS? o Department of Health: Behavioural Risk Survey o Income and Expenditure Survey o Time Use Survey o Community Survey o Urban and Rural Economic Development

Census

o Population census o Agriculture Census Surveys

Deleted: 14/04/2010



Client Satisfaction / Service delivery Surveys

o Public Expenditure Tracking Surveys (PETS) o Cluster Report or Score cards o Citizen report cards

Specialized Surveys

o Where data are non-existent or available data are inadequate

Geographic Information Systems (GIS) / Global Positi oning System (GPS)

Qualitative Surveys

o Key informant interviews o Focus group discussions o Community groups discussions

Population Projections

o HIV/ Aids Projections

The combination of different data sources, such as administrative and survey data are useful for impact evaluation. A unique identifier (although not usually be possible at household level because of issues of confidentiality) is needed to merge the data as the unit of observation of the different data sources usually differs from individual, household data versus community. One could merge at a municipal or small area level. Examples of how data that could be merged as indicated below:

o Poverty maps (census and income-expenditure survey data) o Access to services (GIS/GPS data with survey data) o Child malnutrition (nutrition survey and GIS rainfall data) o Food security (income-expenditure and agriculture production census)

For effective policy development and efficient and effective monitoring of the policy and programme implementation of the overarching PGWC strategic goals and objectives, it has become evident that data needs to be accessible and a common set of data sources for the monitoring of the overarching PGWC strategic goals and objectives is necessary and needs to be existent and readily available for provincial use. The data quality of the common data sources or sets should include the following dimensions:

o Relevance o Methodological soundness o Accuracy o Timeliness o Accessibility o Interpretability o Coherence o Integrity

Deleted: 14/04/2010



4.2.4 DATA QUALITY MANAGEMENT :

En route in pursuance of the above towards quality data, an assessment process, known as an assessment tool, was executed which complemented an earlier audit conducted amongst the provincial departments. In essence, the assessment wanted to check for any gaps and how these are being factored in. In addition, a request on metadata was done in the same assessment to see if it exists. A research and development component was also added to observe capacity within a department to do their own research and if it was aligned to their strategic and annual plans per se. One could determine to some degree whether departments are in a position to supervise and quality assure research that was outsourced or done in-house by them and if their own data was utilised in this regard.

Deleted: 14/04/2010



5 IMPLEMENTATION CONSIDERATIONS 5.1 TRANSFORMATION MANAGEMENT

Change management is needed with regard to the data management. Currently users claim that the data lacks quality, that no common standards exists, many are irrelevant, mostly they are not comparable, many are produced by isolated producers and there are no shared standards and a lack of capacity to produce them. The consequence of this is the lack of reliable evidence to show us how effective our programmers are to plan and to make decisions. In order for the system to be effective, the organisational set-up and management must allow for a seamless flow of information between the departments and various stakeholders with the following prerequisites:

o Understanding the gaps and parameters in which line departments use and construct data sets. Data sets are used and disseminated in a piecemeal manner and users do not necessary see the need to comprehend the data sets coherently.

o Understanding the context and dynamics in which data providers locate themselves. The pilot studies and case studies produced by universities and parastatals in most cases do not reflect a representative sample or full coverage.

o An initial assessment of data users needs. o A critique of the limitations and strengths of commonly used data sets. The

critique could be dual; being the response from the line department concerned and as a data cleaning process.

o Employees capacitated to understand existing data sets e.g. Census, QLFS, PHDI, PERO (report used as a data source), HIV/ Aids Projections

o Monitoring and Evaluation staff capacitated to understand data collection methodology for monitoring

o Monitoring and Evaluation staff to understand the practical processes of monitoring e.g. capturing .cleaning, validation, basic analyses and reporting.

A baseline is the first measurement of an indicator to set the current condition against which future change can be tracked. Importantly, baselines provide the evidence against which decision makers are able to measure subsequent policy, programme, or project performance.

5.2 BUSINESS PROCEDURES

The CD: Monitoring, Evaluation and Review should determine a detailed plan on process and procedures to govern the operation of the system. This should be done taking into consideration the scale of integration, planning and strategies that are involved to make this a reality.

Deleted: 14/04/2010



5.3 TRAINING

The CD e-Innovation in conjunction with CD: Monitoring and Evaluation will provide training manuals for the system.

5.4 QUALITY ASSURANCE

During the Building phase, random testing will be done at each stage completed. Once the whole system is completed, then a thorough testing for the whole system or cycle will be done.

Deleted: 14/04/2010



6 APPENDIX 1: REVISION HISTORY

Version Date Summary of Changes Changes marked Who

V 1 2007-02-002

7 APPENDIX 2: APPROVAL TO PROCEED/SIGN-OFF APPROVAL NOT TO PROCEED

Role Name Date Signature

Project Sponsor

Project Senior User

Project Senior Supplier

Business Analyst

8 APPENDIX 3: DISTRIBUTION

Name Title Telephone No. Business Area

Deleted: 14/04/2010



9 APPENDIX 4: ACRONYMS

Acronym Full Name ANC Ante Natal Care

CD Chief Directorate

CEMIS/FETMIS Central Education Management Information System

DHIS Department of Health Information System

DPLG Department of Provincial and Local Government

FETMIS Further Education and Training Management Information System

GCIS Geographical Communication Information Systems

GDS Growth and Development Strategy

GHS General Household Survey

GWMES Government-wide Monitoring and Evaluation System

HIV/AIDS Human Immuno-deficiency Virus/Acquired Immune Deficiency Syndrome

M & E Monitoring and Evaluation

M and E Monitoring and Evaluation

MDG Millennium Development Goals

MER Monitoring, Evaluation and Review

MERR Monitoring, Evaluation, Review and Reporting

MRC Medical Research Council

PERO Provincial Economic Review and Outlook

PETS Public Expenditure Tracking Surveys

PGDS Provincial Growth and Development Strategy

PSC Public Service Commission

PSP Provincial Strategic Plan

PWMES Provincial-wide Monitoring and Evaluation System

QLFS Quarterly Labour Force Survey

RBM and E Results Based Monitoring and Evaluation

SADHS SA Demographic and Health Surveys

Stats SA Statistics SA

Deleted: 14/04/2010



10 APPENDIX 5: REFERENCES 1. The Presidency of the Republic of South Africa (2005) – Proposal and Implementation

Plan for a Government–wide Monitoring and Evaluation System. The Presidency Policy Unit, Pretoria

2. .Source: Department of the Premier: CD: MER: Current reality of M and E in the

Western Cape province. 2005 3. Sources: Department of the Premier: CD: MER: Current reality of M and E in the

Western Cape province. 2005 p3 and Department of the Premier: CD:MER: Provincial Audit Phase 2, 2007

4. Depart of the Premier (2006 – 2007): Chief Directorate: Monitoring, Evaluation and

Reporting Operational Plan

Deleted: 14/04/2010



ISBN: 978-0-621-39163-3

Deleted: 14/04/2010

The Provincial-wide Monitoring and Evaluation System ... · The planned system is to enable a...

Documents

Transcript of The Provincial-wide Monitoring and Evaluation System ... · The planned system is to enable a...