Learning with participatory monitoring and evaluation in Dir district, northwest frontier province,...

Systems Practice, Vol. 9, No. 2, 1996

Learning with Participatory Monitoring and Evaluation in Dir District, Northwest Frontier Province, Pakistan

S i m o n Be l l t

Received July 21, 1995

This paper is concerned with the development of a monitoring and evaluation system for the second phase of a United Nations Drug Control Programme (UNDCP) in Pakistan and the manner in which participatory structures are developed and built on in a complex development context. Initially the paper introduces the context. The project is concerned with rural development issues as part of a structured programme to substitute opium poppy growing with a range of agricultural and social innovations. During the first phase of the project (up to 1993) the main focus of attention for the project was measured in terms of physical progress with road building, electricity supply, irrigation works and concentration on improved agricultural practices. Phase II has set out to build on this, most specifically in terms of gaining the greater participation of local people in the project activity. The paper describes the monitoring and evaluation (M&E) structure for the project and notes that it is intended that M&E further encourage participation and beneficiary analysis (analysis of project benefits by those who are intended to derive most from the project outputs). One of the primary means set out for achieving this was by inducing the professional staff working in the project to adopt new mind sets, adjusting their focus from being technical experts supplying undoubted technical improvements to being co-operative partners, bringing ideas to the area but equally being willing to work through these ideas and their implications and learn from the communities. The process for the development of the monitoring system is described, and the methodology 'TeamUp' discussed. Links are made to related areas of study--most specifically the literature of Rapid and Participatory Rural Development (RRA and PRA). Finally, the paper discusses the results of the research to date and the likely extension of the ideas to other projects.

KEY WORDS: drug monitoring system; United Nations Drug Control Programme; Pakistan; monitoring and evaluation,

~Systems Department, Open University, Milton Keynes, UK.

129

0894-9859/96/04f)0-0129/$09.50/0 63 1996 Plenum Publishing Corporation

130 Bell

1. BACKGROUND AND CONTEXT

This paper describes a systemic approach to the process of developing a monitoring and evaluation system in a developing country. The area context is the rugged landscape of the Northwest Frontier Province of Pakistan (UNDCP, 1994). The project which this paper deals with is set in Dir District, which is on the Afghanistan border. Dir District Development Project (DDDP) is a UNDCP (United Nations Drug Control Programme) project and is unusual in that it is concerned with rural development as part of a structured programme to reduce criminal/antisocial activity--to substitute opium poppy growing with a range of agricultural and social initiatives. Opium growing in the District is a significant problem [for details on the opiate industry within Pakistan see UNDCP (1994a)], wich contributes to the project's problems--the remoteness of the region, the difficulty in gaining information on illicit behaviour, and the heavily armed population. The core issue for the second phase of the project beginning in 1995--is clearly stated by the UNDCP as follows:

The project aims to encourage the elimination of poppy cultivation from Dir District through focusing on income generation supported by an extensive programme of human resources development, social infrastructural development involving, except where inappropriate the gradual introduction of community participation in project selection, construction and operation and maintenance. (UNDCP, 1994b, p. 12)

At the time when the research described in this paper was ongoing the project was in the early stages of its second phase. The phases of the project demonstrate a change from what Cusworth and Franks (1993) describe as 'traditional blue- print' to 'adaptation' approaches. During the first, 'traditional' phase of the project (1987 to 1993) the main focus of attention was measured in terms of physical progress (e.g., road building, electricity supply, irrigation works, and concentration on improved agricultural practices). Cusworth and Franks describe this approach as follows: 'These are projects in which relatively large amounts of resources are expended in the implementation stage and which normally result in major physical assets' (pp. 8-9). The first phase of the project 'Invested heavily in infrastructure in the poppy growing valleys of the district' (UNDCP, 1994b, p. 8).

Infrastructure development is important for rural development, providing communications and access. However, the focus on 'opening up the area for development,' for physical progress rather than the implicit contribution to social development for the people of Dir, contributed to a lack of focus on this social dimension, resulting in part in 'serious deficiencies in the outreach and content of most of the various extension programmes in the district and extension officers have often been unable to reach farmers' (UNDCP, 1994b, p. 8).

Phase II set out to change the focus of the project, most specifically in terms of gaining the greater participation of local people in the project activity.

Learning with Participatory Monitoring 131

This change in focus conforms to Cusworth and Frank's view of the 'adaptation' approach to project management. It was expected to be 'people based . . . emphasize human or institutional development' (Cusworth and Franks, 1993, p. 9).

The activities described in this paper were a key aspect of the 'people- based' focus of phase II. A concern of the project management and the principle donor's during phase I had been the lack of information on project impact: the effect of the project upon the lives of the people living in the project's primary target area. To improve upon this situation a monitoring and evaluation system was to be planned and to run alongside project activity from the outset. The system to be developed for the monitoring and evaluation (M&E) of the project was to be devised with local project officers and was intended to focus primarily on the impact of phase II on local people as seen through the experience and witness of local people. A core concern for the M&E operation was to make monitoring as participatory as possible, involving project staff and local people.

The background literature provided a number of key observation for the project. Some of the major issues arising are described under three headings: monitoring issues, data gathering, and participatory approaches.

1.1. Monitoring Issues

The DDDP has been running for several years and has been evaluated and appraised in many of its aspects. Considerable time and energy have been focused by aid agencies (such as the Overseas Development Administration of the United Kingdom, ODA), their consultants and UN officials on the nature of the projects activities, outputs, and impacts (e.g., UNFDAC, 1990; Jaqirdar, 1994; ODA, 1993). The outline of physical outputs of Phase II were provided in ODA PEC (94) 37 (ODA, 1994) and in the Project Document (UNDCP, 1994b). In the early stages of the exercise described here it became apparent from a review of these two documents that the expected outputs and therefore impact of the project from the perspective of a major contributing project donor (ODA) and the project management (UNDCP) were essentially similar although each contained distinct points of emphasis. Both agencies had made use of variants of the Logical Framework methodology (of which more shall be said later; for an outline see Fig. 1) in the development of their views of activities, outputs, and impacts and had developed a range of perceptions of the means whereby project impact would be monitored.

Building on the point made above--the Project Document (UNDCP, 1994b, Section B2, p. 10) set out the expected situation at the end of the project. It was felt that this document should provide the ultimate terms of reference for the M&E exercise. In the same document rubrics for M&E are set out on pages 42-43. Of particular importance were sections H1 and H2, which provided a

132 Bell

GOAL

The higher-level objectives towards which the project is expected to contribute

(Mention target groups)

PURPOSE

The effect which is expected to be achieved as the result of the project

O ~ P ~ S

The results that the project management should be able to guarantee

(Mention target groups)

ACTIVITIES

The Activities which have to be undertaken by the project in order to produce the outputs

VERIFIABLE INDICATORS

Meassures (direct or indirect) to verify to what extent the Goal is fulfilled

VERIFIABLE INDICATOR

Measures (direct or indirect) to verify to what extent the purpose is fullfilled


Measures (direct or indirect) to verify to what extent the outputs are produced


Goods and services necessary to undertake Activities

M EA NS OF VERIFICATION

The sources of data necessary to verify status of Goal level indicators

MEANS OF VERIFICATION

The sources of data necessary to verify status of Purpose level indicators

MEANSOF VERIFICATION

The sources of data necessary to verify status of Activity level indicators

MEANSOF VERIFICATION

The sources of data necessary to verify status of Activity level indicators

ASSUMPTIONS

Important events, conditions or decisions necessary for sustaining objectives in the long run

ASSUMPTIONS

Important events, conditions or decisions outside the control of the project which must prevail for the Goal to be obtained

ASSUMPTIONS

Important events, conditions or decisions outside the control of the project necessary for the achievement of the Purpose

ASSUMPTIONS

Important events, conditions or decisions outside the control of the project necessary for the production of the Outputs

Fig. 1. The Logical Framework.

clear focus for monitoring to be both participatory and beneficiary focused: 'both the project and group members [are] to learn from the experience of participatory implementation.' 'Beneficiary monitoring: will be carried out by the PMU (Proj- ect Management Unit) and the Social Organizers in order to determine what the beneficiaries themselves feel about the activities of the project' (p. 42).

In summary, the potential monitoring for the project was to be impact focused--although at the outset there was some confusion in the mind of the


researcher as to what constitutes the expected impact of the project. There was a strong steer for M&E to be participatory and beneficiary orientated.

1.2. Data Gathering

Any M&E exercise is dependent upon some form of data collection exercise. However, a recurrent theme in the literature pertained to the current lack of good data and the poorness of the systems for data collection and monitoring. Lots of problems with sampling and nonsampling errors are provided. For example, both the UNDCP Socio Economic Survey of 1991 and the DDDP Technical Support Unit 'statistical picture' of 1992 indicated problems of nonsampling errors such as illiteracy, lack of household income and expenditure accounts, wide variations in the mode of purchase of consumption goods, agricultural produce not being weighed upon harvest, standing crop being sold, barter of consumer goods, and incorrect statement of incomes. The value and scope for quantitative survey methods in Dir District would appear to be severely limited by the inherent difficulties of the context. Casley and Kumar (1988, p. 6) argued,

It is fruitless to design a data collection operation to reach a widely dispersed sample of respondents with the use of questionnaires and in-depth interviewing techniques if neither the logistic resources to manage such an operation nor the skills to collect and observe accurate data are available.

The authors refer to options and tradeoffs in the design and collection of data. They also refer to the need for both quantitative and qualitative data collection exercises to be employed. This point is born out by Gosling (1995):

Qualitative and quantitative methods do not exclude each other and are often best used together . . . if the main purpose is to build links between agency staff and community members, to transfer skills in information-gathering and analysis, or to pave the way for further development activities, qualitative methods are usually more appropriate. (Gosling, 1995, pp. 44-45)

In summary, the background analysis literature for the project as exemplified in the surveys of 1991 and 1992 indicates that formal, quantitative data collection exercises are hazardous and unreliable. Qualitative techniques might prove more consistent with the context and the beneficiary focus advocated in the Project Document.

1.3. Participatory Approaches

The emphasis in phase II is for participatory project methods--an area of project work which is o f growing interest in development projects (e.g., see Natrajan et al . , 1993; Rajakutty, 1991; Salmen, 1992; Shah and Hardwaj, et

134 Bell

aI., 1993; McPherson, 1994; Webber and Ison, 1995). This approach has already seen some success in Pakistan--most notably in the Aga Khan Rural Support Program (AKRSP). The Word Bank (1990) second interim evaluation states,

The first phase of AKRSP's activity emphasized two primary areas--institutional development and agricultural development. In the first of these, a clear model of institutional change was established, involving the formation of village organizations, collective and participatory decision making, self-help, and savings mobilization. (World Bank, 1990, p. 71)

At another project--the Malakand Fruit and Vegetable Project (MFVP)--rapid and participatory monitoring techniques had been seen to be successful in project methodology (see Sail et al. 1993). Tirmizi (1994) argues that the Malakand project has achieved workable participation in project structure: 'The project is working with a community participation model that demands elaborate mana- gerial and organisational input and coordination from the participants' (p. 33).

Tirmizi notes some scepticism in the DDDP concerning participation (e.g., 1994, p. 3) but argues that local mechanisms already exist within the target area for true partnership. Tirmizi criticises previous reports for a 'lack of specification and concreteness." He argues that they have generally failed to come to grips with the context and have contented themselves in vague description and, thus, 'have not been able to provide a concrete strategy to operationalize the participatory mode apart from specifying staffing requirements for a new tier of externally recruited social organizers in the plural' (p. 3).

Presumably the reports by Weir (1992), endorsed by that of Waddell and Sillitoe (1992), are two of those to which Tirmizi is referring. Weir (1992, p. 8) stresses the difficulties and complexities of participatory approaches and advo- cates a 'low key and partial approach to community participation.'

There appears to be no single view of the potential for participation. Tirmizi appears more upbeat, claiming more detailed local knowledge and advocating the use of existing participatory structures. By contrast, Weir, Waddell, and Sillitoe propose more modest advocacy in the first instance. On the positive side, there is no evidence of local or international consultants strongly dis- agreeing with the participatory approach.

1.4. Summary

A series of conclusions was drawn from the background to the case as suggested here.

�9 The project contains generalised expectations. These, within a common logical framework structure, could be the basis for the M&E exercise and therefore could be reviewed and redesigned to form the background


for the development of more specific Impact Performance Indicators (IPIs), which in turn could form the backbone of the monitoring system.

�9 Quantitative data collection and analysis methods are problematic in the Dir District context. The project literature indicates a range of problems with sampling and nonsampling errors. Further, detailed quantitative analysis takes considerable resourcing. The M&E system under consideration here is likely to be focused on less stringent, qualitative techniques

�9 Participatory methods are also problematic. Local power brokers and cultural factors indicate that detailed knowledge will be required relating to stakeholders in the system and their capacity to adjust/distort qualitative, participatory M&E findings.

2. THE M&E METHODOLOGY

The background information on the project was important in the development of the resulting M&E methodology. Initially the methodology development was undertaken using the soft systems method (see Checkland, 1981; Checkland and Scholes, 1990). Primary issues for the formulation of an appropriate method were seen as being

�9 difficulties in the use of quantitative methods, �9 the project refocus on participatory/beneficiary terms, and �9 equivocation over the use of participatory methods and related difficul-

ties.

Building on this, it can be argued that the M&E methodology for the project needed to reflect the participatory focus set out in the Project Document. Mea- sures were reviewed whereby the methodology could be undertaken in such a way as to encourage further participation and beneficiary analysis (analysis of project benefits by those who are intended to derive most from the project outputs). Given the background of the project in phase I as a technical support operation, one of the primary means suggested for achieving this was by inducing professional staff working in the project to adopt new mind sets, adjusting their focus from being technical experts supplying technical improvements to being co-operative partners, bringing ideas to the area but equally being willing to work through these ideas and their implications and learn from the communities.

Within the core subject of participatory co-operation in M&E design to produce collectable impact performance indicators, four items arose from the literature: stakeholder analysis, impact analysis, management information systems and impact performance indicators (MIS and IPIs), and the resulting selected methodology--TeamUp.

136 Bell

2.1. Stakeholder Analysis

The core item here is set out by Montgomery (1995), where it is argued that stakeholder analysis can enhance the awareness of stakeholders to each other's positions in the project and this in turn can assist in the development of the project. Stakeholder analysis could therefore provide a key for designing a participatory M&E structure--including views of local communities. Following design the resulting structure would be focused on the development of information-gathering procedures on project impact from stakeholder groups. This in turn would provide corroborative information on project impact. Rich pictures and/or Venn diagrams might be used to map out stakeholder impacts. Stake- holder analysis undertaken within the M&E exercise could provide vital information on major groups and major interests and powers. Developed and adapted participatively within the consultancy, it could provide a basis for the M&E project impact analysis. Major outputs from the approach are set out by the ODA Social Development Department (1995, p. 11); these include 'jointly shared implementation and learning; . . . The project has a measurable impact with indicators for measurement identified and agreed by the major stakeholders. '

Stakeholder analysis appears to offer methodology possibilities which are appropriate in the Dir District context. The next issue dealt with was that of the actual measurement of impact.

2.2. Impact Analysis

Project impact can be seen as a measure of the projects level of success in achieving its outputs. From one perspective within the Overseas Development Administration impact reviews are set out clearly within the project framework and identified as being derivable from the Logical Framework. Developing on this theme--the key to impact analysis can again be seen as relating to the logical framework and, most specifically, to the monitoring of outputs to pur- poses. To this end the development of objectively verifiable indicators (OVIs) focused on impact is central.

At this point in the planning for an M&E impact monitoring system a number of cross references were still in mind. For example, the fivefold rapid appraisal methods described by Kumar (1993) could be investigated with a view to developing impact criteria in the first instance. This method might be supplemented/developed in the light of Checkland's soft systems method (in terms of reviewing mindsets of project members and developing project action plans for monitoring procedures) and the Bignell and Fortune (1984) 'failures' approach (for reviewing specific problem areas).

The point made at this stage in the M&E planning process was that simple


and/or single criteria for impact review--be they quantitative or qualitative, participatory or not--should be avoided. Options for methods needed to be kept open until more was known about the context.

2.3. Management Information Systems and Impact Performance Indicators/Objectively Verifiable Indicators

The previous sections take us from participatory project monitoring procedures to impact analysis, and this in turn raises the idea of standard management information systems structures--most specifically the use of entity and attribute modelling (see Bell and Wood-Harper, 1992) in order to provide performance indicators--here described as impact performance indicators (IPIs). Further methodology considerations arise in this context, e.g., explorations in the use of rapid information systems method for MIS development (see Bell and Wood-Harper, 1992).

MIS arising from the use of impact analysis and based upon indicators, stakeholder analysis, the frameworks used by the ODA and the UNDCP for the formulation of their overview of the second phase of the project--all these items indicate that the methodology to be used in the development of the M&E system could be explored within the structure of the TeamUp methodology and software.

2.4. TeamUp Structures

The M&E system which is set out in this paper needs to be able to cope with a wide range of questions, e.g.,

�9 ls there a problem? �9 Who are we looking at the problem (what are the biases and assumptions

which we come cluttered with)? �9 Can we describe the context of the problem? �9 Who are the major stakeholders in the problem context? * What is the timing for the project? �9 Who wants what (donors, recipients)? �9 Is impact occurring? �9 What are the objectives?

The resolving of many of these questions will lead to further formative questions, such as

�9 What do we need to monitor? �9 Who is going to monitor what? �9 How are we going to monitor? �9 When and to what quality are we going to monitor?

138 Bell

The TeamUp approach is one method which was seen as providing potential insights in these areas. In brief the approach comprises a number of stages and the software contains eight tools.

�9 Hiemrchy--a brainstorming tool--the hierarchy of project objectives is set out.

�9 Logical Framework--for project activity, output, purpose, and goal formulation.

�9 Performance plans--workplans for the project--outputs related to yearly plans.

�9 Stakeholders analysis--assigning and assessing project stakeholders. �9 Work Breakdown Structure--assigning tasks to the activity list. �9 Organisational responsibility chart--assigning authority to project staff. �9 Gantt Chart-scheduling tasks to dates. �9 Project budget-as it sounds.

See Team Technologies (1995). The work of Thompson (see Thompson, no date, Thompson and Chudoba,

1994) indicated that the Logical Framework structure, in particular, could offer a methodology which was systemic, could combine existing plans, and could accommodate participatory concepts.

2.5. Summary

The methods section indicates two items.

�9 Stakeholder analysis of the monitoring team needs to occur first. This can be linked to personality analysis and will need to result in the participative development of feasible monitoring structures--based upon the TeamUp experience but including other methodologies--such as the soft systems approach.

�9 Impact analysis is also containable within the TeamUp structure but might be developed in IPI format by use of MIS development tools such as the rapid approach adopted by Bell and Wood-Harper (1992). IPIs could be reviewed in the first instance making use of standard participatory appraisal tools.

3. THE DEVELOPMENT OF THE M&E SYSTEM--EFFECTIVE USE OF TEAMUP?

The bulk of this section describes how the M&E system was designed and the fundamental structures within this design. The description is kept to a minimum-the interest of this paper being more centered on the implications of this


Compare different Iogframes Agree on Goal

Indicate how IPIs ~ "' ~ Goal OVl and MOV feedback into ~ managment & / r e p o r t s /

~ Agree on Purpose(s), Indicate how IPIs are ~ ~ Pumose OVI and MOV ie be used and evaluated / Logframe ~ ,~,

[ M&E design ,j, procees ~ Structure output ~ IPls, provide details \ of techniques (including\ ,/Agree on Outputs SWOC) ~ . / Set out relevant IPIs

~ . / Give details of MOVs down to the indicatorsAgree evaluation ~ . . . . . . . _ ~ ~ level of general techniques

to be applied Agree on Financial Agree on PIs Activities and outline

of Progress PIs

At this stage this is very preliminary

Fig. 2. The M&E design process.

type of system rather than the laborious explanation of technical detail. Figure 2 describes the nature of the M&E design process.

The M&E design process was broken down into five major blocks of activity:

* Initial review-including stakeholder analysis �9 Actual and potential impact performance indicator (IPI) review �9 Detailed IPI identification �9 Participative modelling of monitoring procedure �9 Review

These activities were undertaken over a 26-day period in the Spring of 1995. In the following subsections each activity is briefly described.

3.1. Activity Item 1--Initial Review

Stakeholder analysis was undertaken over the first 8 days of the design procedure (along with other items) and included a rich picture analysis to assist in problem issue identification. The stakeholder analysis followed the TeamUp format, focusing on the stakeholders for the M&E aspect of the project (rather than the project in general). Stakeholders views were taken from donor groups, local implementation agencies, beneficiaries, and local leaders. Due to time

140 Bell

constraints it was not possible to interview as many minor stakeholders as would have been most desirable. However, the analysis does represent a wide range of views on the project performance. The range did provide some useful conclusions. Themes were picked up during the stakeholder analysis which were to guide the following design process. As would be expected in a project of this complexity there were a number of different steers on project M&E activity. These were specified as the need for performance impact information for MIS, a thorough review of the feasibility of the system, focus on problems with regularity and frequency of reporting, some form of financial information systems-including uniform procedures and methods, the need for the morale of project staff to be monitored and taken due account of, and the need for the project to be accountable to local people.

Following stakeholder analysis an extensive review of the monitoring capability of the project was undertaken. This activity was centered on discussions with officers involved in monitoring procedures. The existing monitoring capability of the project was centred in the recently created Project Monitoring Co- ordination Section. This section comprises two staff--a Monitoring Economist and a Financial Systems/Audit Officer. The Section was new and both officers were only beginning to organise their role. Neither of the officers appointed has had any training of any kind in the procedures and requirements of monitoring and evaluation. Three primary items arose from the review.

(i) The existing staff have not had specific training in M&E procedures. (ii) The ad hoc nature of much-requested reporting made the formulation

of structure in reporting difficult to maintain. It also militated against the M&E Section having a clear internal Workplan.

(iii) Current items monitored and reported on related primarily to financial and physical progress. There was no current emphasis on participatory 'impact' monitoring.

The next activity undertaken was a review of existing data. This review related to three primary sources of data: the 1991 baseline survey, the 1992 'Socioeconomic Profile of Poppy Growing Valleys in Dir District,' and 6-monthly 'Progress Reports' produced by the project. The existing data for the project indicated two major features.

(i) The 1991 baseline survey, followed and supplemented by the 1992 Socioeconomic profile was flawed--as indicated in Section 2. Quanti- tative data analysis is therefore problematic.

(ii) The semiannual progress reports were seen to be a basis for progress monitoring and could possibly provide a medium for impact analysis, showing in a 6-monthly format the way in which the various sectors of the project are meeting the outputs expected in the Project Document.


The review of existing data was followed by a review of locally identified critical features. In this stage of Activity 1 the basis for monitoring was reviewed. The stakeholder analysis provided the main steer on this aspect, however, a contributing element was the objectives of the UNDCP Project document. Crit- ical features were derived from donor and local management views and concerns. These were summarised as follows.

(i) The 20 project outputs as set out in the Project Document should be the first basis for impact monitoring.

(ii) The M&E section also needs to concern itself with a provisional financial monitoring system.

(iii) The feasibility of quarterly reporting in some areas requires examina- tion.

So far the analysis had worked with local project staff and the design had integrated donor policy with local views and priorities. The analysis now went on to review local perceptions on project shortcomings and logistical issues in more depth. These points were derived primarily during the interviewing for the stakeholder analysis. They can be listed as follows (no order of precedence is intended).

(i) Project staff have very limited time. They are interested primarily in the execution of project activities and their effectiveness. The minimum time should be allotted to extra monitoring activity.

(ii) The Monitoring Co-ordination Unit is underpowered for the monitoring task allotted to it. Training and resources are key issues.

(iii) Several areas of project activity are poorly supported in terms of staffing. There is a severe lag in terms of staff recruitment partly due to the remoteness of Dir.

(iv) Social organisation and staff concerned with the issue of Women in Development were particularly poorly supported in terms of staff num- bers. It is very difficult to recruit women to work in Dir because of the limitations put upon them by the traditional Islamic culture of the area.

(v) Participation is a new issue in the area. It must be imported (as a concept for work activity) gently, bringing the local communities along with it. Much has been made locally of the success of other projects (most specifically the Aga Khan Rural Support Programme; AKRSP) at participatory methods. There is active discussion in the area concerning the 'replication' of AKRSP techniques. Stakeholder analysis indicated that caution would be required in any such activity (as suggested by Weir, 1992; Waddle and Sillitoe, 1992) and that context-specific factors (such as the potential manipulation of participatory activities by domi- nant social elites) needed to be considered in any such replication

142 Bell

attempt. A concern of the project in its second phase is that participation should support those at present underrepresented by traditional social forums.

Finally, the first activity concerned itself with the identification of local staff to participate in following analysis. This activity was again derived primarily from the stakeholder analysis (informative for most of Activity 1). Sug- gested key staff for the monitoring tasks were recommended and agreed.

Activity 1 demonstrated evidence of interest in project progress locally. This was an informative process for the development of the M&E indicators which followed.

3.2. Activity Item 2--Review of Actual and Potential IPIs

The first aspect of Activity 2 was a comparison of existing project indicators with potential indicators. For this the Strengths, Weaknesses, Opportunities, and Constraints (SWOC) analysis was used. This activity involved a wide- ranging discussion with key project staff, most specifically the STA (Senior Technical Adviser). The focus for the discussion was the development of a Logical Framework, containing output IPIs which the STA considered could generally be produced every 6-months and which would provide stakeholders inside and outside the project with an indication of 'impact' in the various core areas of project activity. Each potential indicator was reviewed in terms of its SWOC. This provided those involved with the monitoring design process with insights into the validity of the indicator and means of verification (MOV). Following from this the preliminary Logical Framework was reviewed and altered. There followed a feasibility analysis.

It is worth noting at this point that it was discovered during SWOC that many of the 31 IPIs could be termed 'virtuous measures.' The virtue arises from the observation that they very often encouraged regular, structured reporting and documentation from staff where this was previously assumed in the project. Further, many indicators were virtuous in that they required local staff to become trained in and familiar with participatory techniques. It was also considered that the Means of Verification would encourage good practice by project staff and this would in turn lead to higher impact in project activities and the related recording of this impact--a virtuous, "reinforcing" (Senge, et al., 1994, p. 116) cycle (see Fig. 3).

The feasibility analysis of potential monitoring system dealt with the assessment of feasibility of each indicator as initially measured by the SWOC analysis provided in the previous section. Feasibility included who would be doing what and when and the degree of confidence felt by project staff in the likelihood of the MOVs producing a good indicator of impact. As with the SWOC this was


Develop IPIs

t ~ ~'~ ~

good T p r a c t i ~

to prod uce~.....~ ~ and

providing good documentation

Fig. 3. The virtuous cycle of IPI development.

learning participatory methods

a rich opportunity to reflect on the nature of the Logical Framework and discuss issues with local staff. Further alterations were made at this time.

The next item covered was the assessment of local capability, i.e., the extent to which the monitoring section of the project has access to necessary skills and techniques, as well as basic data, in order to be able to fulfil the requirements set out in deriving the IPIs.

This stage revealed the extent to which the monitoring process would need to be supported by training and development of local staff. An important item noted clearly at this stage for the first time was the lack of 'participative analysis' skills among staff and the major role which the project's Senior Social Organiser would be expected to play in developing awareness and practical skills in this area among fellow consultant staff. Several other officers were seen to be sig- nificantly loaded by monitoring activity and plans to deal with this were discussed.

3.3. Activity Item 3--Detailed IPI Identification Having specified 31 indicators across the project (dealing with impact on

the lives of those living in the target area ranging across the full extent of project activities) it was necessary clearly to identify those indicators (to be either quantitatively or qualitatively generated) in conformity with three criteria.

(i) Identified as being of high value in the Project Framework and overall objectives.

144 Bell

(ii) Identified as being of importance in discussion with local staff. (iii) Identified as being viable and sustainable in terms of data gathering

realities.

This activity provided an opportunity for reflection on the monitoring process so far undertaken. The indicators identified to date were reappraised and the layout as given in the Logical Framework was confirmed.

The next item was to look at MOV questionnaire design and sampling procedures if required and appropriate. This section was expanded to cover the details of the manner in which each IPI was to be produced. To this extent the design of the proformas was set out as well as the basics of the sampling and interview procedures to be followed where these were not already widely known.

It was now appropriate to review the outline management information system (MIS). The analysis to date had concentrated on the design of an impact performance indicator M&E system to assist project management in knowing what impact the project was having on the lives of people in Dir. However, in Section 3.1 it was found that there was a developing requirement for other forms of information which might be included in an M&E Management Information System (MIS).

MIS is seen here as encompassing three distinct information products-- impact, progress, and financial indicators. The outline for local level MIS was restricted by time to the development of the impact context. Consideration was provided at this time to what management would do with the IPIs when produced.

3.4. Activity Item 4--Participative Modelling of Monitoring Procedure

In the previous sections it has been described how analysis and design had produced the basis of an impact analysis system combining participation in its operation and reflection on internal issues such as project staff morale. Now a participative approach was adopted to check over the range of PIs which had been produced so far, for both the initial system (pertaining to IPIs) and later the Financial Performance Indicators and Progress Performance Indicators (PPIs). The M&E PI system was set out as covering three phases:

Phase 1, Impact Performance Indicator production; Phase 2, Financial Performance Indicator production; and Phase 3, Progress Performance Indicator production.

An area which prompted special consideration was the demarcation of data collection and analysis methods. The participative approach adopted was used to set out the range of methods to be applied in the monitoring process. Of the total IPIs, 15 were quantitative, 9 were qualitative, and 7 combined both quantitative and qualitative techniques. Many of the IPIs made use of rapid and


participatory appraisal methods [such as those advocated by Gosling (1995), Kumar (1993), and Chambers (1992)]. Overall, it was thought by members of the project team that the IPI monitoring process showed a diverse range of techniques. This, it was thought, should help to ensure that indicators repre- sented the various levels of the project, the various ways in which these activities could be appreciated or experienced, and the internal dynamics of the project (e.g., staff interaction and morale, etc.). The overall methodology development for monitoring and evaluation of the project was concerned with the means by which monitoring activity was to be contained in one function, the methodo- logical application of that function, and the manner in which monitoring products would feed back to management, and likely management activity plans arising from the scenarios portrayed in the IPIs.

Each IPI was provided with three potential conclusions--satisfactory and below and above satisfactory. The interpretation of these needed to be undertaken carefully and needed to recognise that no single IPI result would be definitive or would result in a specific change in current activity by itself.

3.5. Activity Item 5--Review

This activity was focused on the results of a half-day workshop held with project staff in order to review the results of the M&E analysis and design to date. The initial Logical Framework, setting out the details of the IPI system, was distributed.

The main concerns of those present were as follows.

(i) The system should generate no more extra work than is absolutely necessary and this work should not have a negative impact upon phase II implementation.

(ii) Monitoring should be internally reflective of the project organisation-- reviewing management structures and, where possible, identifying management bottlenecks. Monitoring should provide internal project learning.

(iii) Beneficiary analysis is in its infancy in the project. The successful development of participative techniques in monitoring of project impact would require both the planned inputs of the Senior Social Organiser (in terms of training other project consultants) and the wholehearted participation of project consultants in the activity.

3.6. Summary

The analysis set out above resulted in the production of a Logical Frame- work setting out the M&E goal, purpose, outputs, and activities. The structure

146 Bell

included tasks for those involved and a review of the strengths, weaknesses, opportunities, and constraints of each performance indicator and a measure of expected feasibility. The Framework was produced in a participative manner with local staff and the resulting M&E system was seen to produce indicators consistent with management information systems design.

4. RESEARCH RESULTS TO DATE AND TENTATIVE CONCLUSIONS

The project for M&E design described in this paper is entering its implementation stage and it is problematic to draw too many conclusions at the present time. Nevertheless, some generalised observations are possible.

�9 TeamUp proved to be eclectic in its application--containing both quantitative and qualitative measurement capability and containing in the 'Assumptions' column the capability to integrate 'soft' issues as well as 'hard' project data and activity.

�9 The TeamUp methodology included in a monitoring project offers a rich range of points of analysis and comparison, for example, the use of verifiable indicators of project outputs as a basis for an impact monitoring system and the use of indicators of performance related to project activities as a performance monitoring system. Figure 4 demonstrates the manner in which the four levels of the project can each contribute to a specific point in the M&E process.

�9 The 'indicators' were seen to be potentially either qualitative or quantitative, thus allowing for participatory, qualitative items as well as those which are more formal.

�9 The inclusion of stakeholder analysis in TeamUp was useful in developing the monitoring system around the views of major beneficiaries and recipients as well as the project actors and donor agencies in the context.

�9 The TeamUp approach was seen to be systemic in that it provided a sound basis for a sustainable monitoring system and the resulting plan for the system conformed to all the major features expected of a formal system (e.g., an environment for the system with a boundary, inputs and outputs, open or closed, goal seeking, purposeful, and with some form of control expressed).

�9 Although the context in the case described was very specific, it would appear that the TeamUp approach offers considerable opportunity for use as a planning and monitoring tool in a wide range of situations.

To conclude, TeamUp as a systemic M&E development methodology appears to conform to the requirements of projects which are attempting to provide

Learning w i t h P a r t i c i p a t o r y M o n i t o r i n g 147

Goal

t leads to

Purpose

T leads to

Outputs

l leads to

Activities

Fig. 4.

Indicators of progress to goal

Means of verification

Indicators of objectives evaluation

help~ovide f

Impact perormance indicators

~ ide

Progress performance indicators




The Logical Framework as a basis for M&E.

M&E function

Poject contribution to wider goals

Evaluation - post project

Measures of project impact - six monthly

Review of project activity - daily monitoring

participative and beneficiary orientations. Further issues and description will follow the implementation of the initial design and the inevitable changes that are made in the light of experience.

REFERENCES

Bell, S., and Wood-Harper, A. T. (1992). Rapid Information Systems Development--Systems Anal- ysis and Design in an Imperfect World, McGraw-Hill, Maidenhead.

Bignell, V., and Fortune, J. (1984). Understanding Systems Failures, Manchester University Press, Manchester.

Casley, D., and Kumar, K. (1988). The Collection, Analysis and Use of Monitoring and Evaluation Data, World Bank, Johns Hopkins University Press, Baltimore.

Casley, D., and Kumar, K. (1988). Project Monitoring and Evaluation in Agriculture, World Bank, John Hopkins University Press, Baltimore.

Chambers, R. (1992). Rural Appraisal: Rapid, Relaxed and Participatory, IDS Discussion Paper 311.

148 Bell

Checkland, P. (1981). Systems Thinking, Systems Practice, Wiley, Chichester. Checkland, P., and Scholes, J. (1990). Soft Systems Methodology in Action, Wiley, Chichester. Cusworth, J., and Franks, T. (1993). Managing Projects in Developing Countries, Longman Sci-

entific and Technical, Harlow. DDDP Technical Support Unit (1992). Socio economic profile of poppy growing valleys in Dir

District: A statistical picture--June 1992, Timergara. Gosling, L., and Edwards, M. (1995). Assessment, Monitoring, Review and Evaluation Toolkit.

Save the Children Fund, London. Jaqirdar, H. (1994). Indicative economic and financial appraisal of Dir District Development project

phase lI--Drafl final report, July. Kumar, K. (1993). Rapid Appraisal Methods, World Bank, Washington, DC. McPherson, S. (1994). Participatory Monitoring and Evaluation Abstracts, IDS. Montgomery, R. (1995). Guidance Notes on Stakeholder Analysis for Aid Projects and Programmes,

Centre for Development Studies, Swansea. Natrajan, L., et al. (1993). Comparative study of sample survey and participatory rural appraisal

methodologies with special reference to evaluation of National Programme on Improved Chu- lah, National Council of Applied Economic Research, New Delhi.

ODA (1993). Dir District Development Phase II Appraisal Mission, Final Report, ODA, London, Nov.

ODA (1994). Project and Evaluation Committee (1994) 37 Dir District Development Project Phase II Submission, ODA, London.

ODA Social Development Department (1995). Enhancing Stakeholder Participation in Aid Activi- ties, ODA, London.

Rajakutty, S. (1991). Peoples participation in monitoring and evaluation of rural development pro- grams: Concepts and approaches. J. Rural Dev. (India) 10(1), 35-53.

Sail, S., Young, S., Shah, T. A., and Junaid, M. (1993). Malakand Fruit and Vegetable Devel- opment Project, Information gathered during phase 2, Saidu Sharif.

Salmen, L. (1992). Beneficiary Assessment, an Approach Described, World Bank Discussion Paper 31, Washington, DC.

Senge, P., Kleiner, A., Roberts, C., Ross, E. and Smith, B. (1994). The Fifth Discipline Fieldbook, Nicholas Brealey Publishing, London.

Shah, P., Hardwaj, G., et al. (1993). Gujerat, India: Participatory monitoring Rural Extens. Bull. 1, 34--37.

Team Technologies Inc. (1995). TeamUp 2.0, Team Technologies, Inc., Chantilly. Thompson, M. (no date) A TeamUp Case Study: Agriculture project design. Team Technologies

Inc., Chantilly. Thompson, M., and Chudoba, R. (1994). Case Study Municipal and Regional Planning in Northern

Bohemia, Czech Republic: A Participatory Approach, World Bank Report, Washington, DC. Tirmizi, J. (1994). Aspects of Social Organisation and the Operationalisation of a Participatory

Approach Within the Dir District Development Project, report submitted to UNDCP, Technical Support Unit, Timergara.

UNDCP (1991). Socio Economic Survey of Poppy Growing in Dir District, 4 vols., UNDCP, Vienna.

UNDCP (1994a). The Illicit Opiate Industry of Pakistan, Summary, UNDCP, Islamabad. UNDCP (1994b). DDDP Phase II Project Document--UNDCP--Dir District Development Project,

UNDCP, Vienna. UNFDAC (1990). Report of the In-Depth Evaluation Mission Carried out in June--July, Vols. 1-

5, Vienna. Waddell, R., and Sillitoe, P. (1992). Report of the Mission on Sustainable Development, ODA,

London.


Webber, L., and Ison, R. (1995). Participatory rural appraisal design: Conceptual and process issues. Agri. Syst., 47, 107-131.

Weir, A. (1992). The Potential for Community Participation, UNDCP, Vienna. World Bank (1990). The Aga Khan Rural Support Program in Pakistan. A Second Interim Evalu-

ation, World Bank, Washington, DC.

Learning with participatory monitoring and evaluation in Dir district, northwest frontier province,...

Documents

Transcript of Learning with participatory monitoring and evaluation in Dir district, northwest frontier province,...