Training Report and Manual

PROJECT MANAGEMENT AND MONITORING &

EVALUATION

HANOI, VIETNAM

26/11/2012-30/11/2012

CONTENTS

Introduction...........................................................................................................................................1

Planning and timing...........................................................................................................................1

The trainees.......................................................................................................................................2

The trainers.......................................................................................................................................3

Training Manual.....................................................................................................................................4

Lecture 1: Project management............................................................................................................4

1. Defining Project Management...................................................................................................4

2. The importance of project management...................................................................................4

3. The role of the project manager................................................................................................4

4. Keys to successful project..........................................................................................................5

5. Managing the contractor...........................................................................................................6

6. Managing the project................................................................................................................6

Lecture 2: Result based management...................................................................................................7

1. What is Results-based management (RBM)...............................................................................7

2. The purpose and benefits of RBM.............................................................................................8

3. RBM involves three major steps................................................................................................8

4. The Results’ Chain......................................................................................................................8

Lecture 3: Basic M&E concepts............................................................................................................12

1. Monitoring...............................................................................................................................12

2. Evaluation................................................................................................................................12

3. Important terms......................................................................................................................13

Lecture 4: M&E system........................................................................................................................16

1. What is M&E system?..............................................................................................................16

2. How to set up the M&E system...................................................................................................17

3. M&E Plan.....................................................................................................................................21

3.1 The structure of the monitoring plan.....................................................................................21

3.2 The structure of the evaluation plan......................................................................................22

3.3 M&E and project logic frame.................................................................................................23

3.4 Detailing M&E plan................................................................................................................24

3.5 Using the M&E Matrix for detailed planning.........................................................................25

3.6 Indicative contents for an M&E Plan......................................................................................26

4. Data collection methods..............................................................................................................29

5. Gender and M&E.........................................................................................................................31

6. M&E implementation responsibilities.........................................................................................32

Lecture 5: Gender mainstreaming in development projects...............................................................35

1. Basic definition when conducting gender analyzing................................................................35

2. Gender mainstreaming in development project......................................................................36

2.1 What is gender mainstreaming?......................................................................................36

2.2 Why do we need gender mainstreaming?.......................................................................37

2.3 Objectives of gender mainstreaming in projects.............................................................37

2.4 Tools and strategies for gender mainstreaming..............................................................37

2.5 Gender analysis................................................................................................................38

2.6 What to expect in a successful gender mainstreaming project.......................................39

2.7 Basic principles in gender mainstreaming........................................................................39

Lecture 6-7: Measuring and reporting first and second level of results..............................................40

1. Overview..................................................................................................................................40

2. Why to measure and report the first and second level results?..............................................40

3. Collecting information and reporting first and second level results........................................41

4. Measuring first level results.....................................................................................................42

5. Measuring and reporting second level results.........................................................................44

Lecture 8: Steps to conduct a survey...................................................................................................47

1. Step 1: Define objectives of the survey........................................................................................47

2. Step 2: Assess the resources....................................................................................................47

3. Step 3: Determine methods.....................................................................................................47

4. Step 4: Designing questionnaire, interview guide and pilot.....................................................48

5. Step 5: Selecting the sample....................................................................................................49

6. Step 6: Enumerators’ training and field testing of the questionnaire......................................49

7. Step 7: Logistical planning.......................................................................................................50

8. Step 8: Data collection.................................................................................................................51

9. Step 9: Data entry........................................................................................................................51

10. Step 10: Data analysis................................................................................................................51

11. Step 11: Report writing..............................................................................................................51

12. Step 12: Communication – Dissemination.................................................................................52

Lecture 9: Questionnaire design..........................................................................................................53

1. What are questionnaire?.........................................................................................................53

2. The importance of questionnaire................................................................................................53

3. Steps to design questionnaire......................................................................................................53

3.1 Study project documents.......................................................................................................53

3.2 Identify the indicators, goals..................................................................................................54

3.3 Study the template questionnaire of IFAD.............................................................................54

3.4 Develop new questions, gender questions............................................................................54

3.5 Structuring the questions......................................................................................................54

3.6 Questionnaire presentation...................................................................................................54

4. Principles for questionnaire design..............................................................................................55

5. Questionnaire piloting.................................................................................................................56

6. Summary of steps to design the Questionnaire...........................................................................56

Lecture 10: Sampling method..............................................................................................................57

1. Definitions...................................................................................................................................57

1.1 Population.............................................................................................................................57

1.2 Sample...................................................................................................................................57

1.3 Sampling frame......................................................................................................................57

1.4 Sample size............................................................................................................................57

2. Overview sampling method.........................................................................................................57

2.1 Random sampling method.....................................................................................................57

2.2 Non-random sampling method..............................................................................................58

3. Sampling method in project funded by IFAD...........................................................................59

3.1 Stage 1: Select villages...........................................................................................................59

3.2 Stage 2: Select households....................................................................................................61

Lecture 11: Introducing the Annual Outcome Survey..........................................................................63

1. What is the annual outcome survey?..........................................................................................63

2. How and when to conduct the Annual Outcome survey?...........................................................63

3. Steps to conduct the AOS............................................................................................................63

3.1 Step 1: Designing questionnaire, interview guide and pilot...................................................63

3.2 Step 2: Sampling..............................................................................................................64

3.3 Step 3: Enumerators’ training and field testing of the questionnaire....................................64

3.4 Step 4: Logistical planning......................................................................................................65

3.5 Step 5: Data collection...........................................................................................................65

3.6 Step 6: Data entry..................................................................................................................65

3.7 Step 7: Data analysis..............................................................................................................65

3.8 Step 8: Report writing............................................................................................................65

Lecture 12: RIMS and RIMS+ baseline and end-line survey.................................................................67

1. Introduction to RIMS and RIMS+ survey..................................................................................67

1.1 Background on RIMS..............................................................................................................67

1.2 Changes in RIMS+..................................................................................................................67

2. How to conduct RIMS and RIMS+ survey.................................................................................70

Lecture 13: Key informant interview...................................................................................................72

1. Definition of key informant interviews....................................................................................72

2. Classification of key informant interviews...............................................................................72

3. Steps to conduct key informant interviews.............................................................................73

3.1 Identify the topics that need in-depth information...............................................................73

3.2 Identify the key informants....................................................................................................74

3.3 Develop the questions by topics............................................................................................74

3.4 Prepare the interview guide..................................................................................................74

3.5 Pre-test and finalize the questions or interview guide..........................................................75

3.6 Conduct the interview...........................................................................................................75

3.7 Analyze and write report.......................................................................................................76

Lecture 14: Focus group discussion.....................................................................................................78

1. Overview of focus group discussion............................................................................................78

2. Steps to facilitate a focus group discussion.................................................................................78

2.1 Identify topics and information needed for each topic..........................................................78

2.2 Prepare guide for discussion..................................................................................................79

2.3 Select participants...........................................................................................................79

2.4 Conduct FGD..........................................................................................................................80

2.5 Analyze results and write report............................................................................................81

LIST OF TABLE

Table 1: Result Chain.................................................................................................................9Table 2: Example of Result chain- IFAD’s approach...............................................................10Table 3: Example of Result chain- CIDA’s approach.............................................................10Table 4: Monitoring and evaluation.........................................................................................12Table 5: M&E tasks.................................................................................................................16Table 6: Critical Reflection Schedule for a project..................................................................19Table 7: Questions to guide the detailed planning of M&E conditions and capacities...........20Table 8: The structure of an evaluation plan............................................................................22Table 9: Task needed when detailing the M&E plan based on a project appraisal report.......24Table 10: Indicative contents for an M&E operational Plan....................................................27Table 11: Examples of multi-purpose M&E methods.............................................................29Table 12: Comparison of quantitative and qualitative methods...............................................30Table 13: Discriminate between sex and gender......................................................................35Table 14: Example for 1st level results.....................................................................................41Table 15: Example for 2nd level results....................................................................................41Table 16: Example for indicator about rehabilitation of infrastructure and facilities..............42Table 17: Example for indicators about training......................................................................43Table 18: Example for indicators about group formation/strengthening.................................43Table 19: Example of reporting first level...............................................................................44Table 20: Rating scale for second level result.........................................................................44Table 21: List of performance questions for each factor.........................................................45Table 22: Framework of factors and guiding performance questions......................................46Table 23: Advantages and disadvantages of methods of administering survey questions......48Table 24: Overview of changes in RIMS+..............................................................................67Table 25: Information in RIMS and RIMS+ questionnaire.....................................................68Table 26: Example for measuring impact of project................................................................70Table 27: Advantages and disadvantages of unstructured and semi-structured interview......73

LIST OF FIGURE

Figure 1: Project process............................................................................................................9Figure 2: Project Results-Chain...............................................................................................10Figure 3: The use of control group...........................................................................................61

LIST OF BOX

Box 1: Five key evaluation questions......................................................................................13Box 2: The basic performance questions per level of the objective hierarchy........................18Box 3: Examples of performance questions.............................................................................18

Introduction

IFAD has been working for and with the poorest people in Vietnam, including ethnic

minorities, small-scale farmers and households headed by women. At present, IFAD is

working in ten provinces in Viet Nam (9 projects), and some other projects will be

implemented soon. In order to assess its progress against implementation plans (monitoring),

and to assess the outcomes and impact it has achieved (evaluation), the project’s monitoring

and evaluation (M&E) system is extremely important. The purpose of monitoring and

evaluation is to obtain accurate information that will be used by decision-makers in the

project to improve and ensure proper use of resources as well as to maximize the impact

achieved from these resources.

However, there is a fact that most of IFAD project management staff in Viet Nam has

limited exposure to IFAD M&E system. Therefore, building and improving capacity for IFAD

project’s management staff is extremely important. The management staff needs to be equipped

with knowledge and practices on project management and M&E, in order to organize and

manage resources so the project is completed within defined scope, quality, time, and cost

constraints.

Development and Policy Research Center (DEPOCEN) coordinated with APMAS and

IFAD country office, Viet Nam to organize training on Project management and M&E in

Hanoi to equip the IFAD Project Managers and M&E officers in Vietnam with knowledge

and practices on M&E.

Planning and timing The duration of training course was 5 days, from 26 to 30 November, 2012 in Hanoi. The

course was hence expected to provide participants the following knowledge and skills:

• Project management

• M&E, basic concept and methodology, how to set up M&E system, M&E plan etc.

• Mainstream gender in development project and in M&E strategy

• Measuring and report first, second and third level results

• Sampling method

• Methodology of collecting qualitative information including key informant interview

and focus group discussion

The course was designed to provide trainees with both theory and opportunities to

practice. In every section, participants were motivated to raise questions, comments, sharing

experience on specific themes as well as to engage in exercises in which they have the chance

Page 1

to apply theory to solve practical problems. In addition, participants listened to experiences of

Ninh Thuan project staff in conducting RIMS+ baseline survey and experiences of Tuyen

Quang project staff in conducting annual outcome survey.

Table 1: Project management and M&E training program Date Time Content

26/11/2012(Monday)

Morning 1. Project Management

Afternoon2. Results Based Management3. M&E Basic concepts4. M&E system

27/11/2012(Tuesday)

Morning 4. M&E system (cont.)

Afternoon5.Gender mainstreaming in development project and gender sensitization of M&E strategy

28/11/2012(Wednesday)

Morning6. Measuring and reporting 1st level result7. Measuring and reporting 2st level result

Afternoon8. Steps to conduct survey9. Questionnaire design

29/11/2012(Thursday)

Morning

10. Sampling methodology11. Annual outcome survey12. Annual outcome survey – (presented by representative

from Tuyen Quang)

Afternoon

13. Measuring and reporting 3rd level result14. RIMS+ baseline survey (presented by representative from

Ninh Thuan)15. RIMS+ end-line survey (presented by representative from

Ha Tinh)30/11/2012

(Friday)Morning 16. Key informant interview

Afternoon 17. Focus group discussion

The trainees

Total of 43 trainees participated in the training course, including project staffs from Cao

Bang (05), Bac Kan (07), Ha Tinh (03), Quang Binh (02), Tuyen Quang (05), Ninh Thuan

(03), Gia Lai (07), Dak Nong (06), Ben Tre (05).

Most participants were enthusiastic and involved in the training course with good

attitude. They actively engaged in the lectures by giving questions and examples and sharing

their experiences that made the lesson attractive and exciting.

After the training, all trainees were asked for an assessment on the training quality and

effectiveness. Through feedback and comments of trainees on training contents and teaching

strategy, it will be used by trainers to improve the quality of the next courses in the future.

Page 2

The trainers

Trainers participating in delivering the training included trainers from DEPOCEN: Ms. Nguyen

Thi Phuong Mai - an adviser on M&E, conducting survey, in-depth interviews and questionnaire

design; Ms. Tran Thanh Thuy, Ms. Nguyen Thi Huyen and Ms. Pho Thanh Huong. Ms. Hoang Hong

Hanh, a gender expert contracted by APMAS, was invited to give lecture on gender mainstreaming.

Trainers’ responsibility, enthusiasm and friendliness created a favorable environment for learners’

participation.

Table 2: Trainers participating in the training Full name Organization Tasks

1. Ms. Nguyen Thi Phuong Mai DEPOCEN

- Project Management- Results Based Management- M&E Basic concepts- M&E system- Measuring and reporting first level result- Measuring and reporting second level result- Measuring and reporting third level result- Annual Outcome Survey

2. Ms. Hoang Hong Hanh Gender expert Gender mainstreaming in development project and gender sensitization of M&E strategy

3. Mrs. Tran Thanh Thuy DEPOCEN - Sampling methodology- Key informant interview

4. Ms. Nguyen Thi Huyen DEPOCEN - Steps to conduct survey

5. Ms. Pho Thanh Huong DEPOCEN - Questionnaire design

6. Mr. Le Thanh Bien TNSP TUYEN QUANG Annual Outcome survey in Tuyen Quang

7. Mr. Truong Truong Han

TNSP NINH THUAN RIMS+ baseline survey in Ninh Thuan

8. Mr. Le Van Hiep IMPP HA TINH RIMS+ end-line survey in Ha Tinh

Page 3

Training ManualLecture 1: Project managementWhat is a project?A general of related policies, activities and expenditures are designed to achieve certain objectives in a given time. PM Triple ConstraintsMain challenges of project management:

Achieve all expected goals within a certain scope (workloads and technical requirements), to ensure project benefits reach to all beneficiaries including women, men, girls and boys

Complete in time (performance progress) Use in given budget (investment) Meet all requirements (quality) and expectations.

1. Defining Project Managemento Project management is the application of knowledge, skills, tools and techniques to

project activities to meet project requirementso Organizing and managing resources so the project is completed within defined scope,

quality, time and cost constraints2. The importance of project management

Why Projects Failo Lack of senior management commitmento Inadequate project planning (budget, schedule, scope, etc.)o Absence of community participatory o New or unfamiliar technologyo Lack of defined, clear, or concise requirements

Why Projects Succeedo Sound project management processeso Senior management commitmento Detailed requirementso Realistic scheduleo Good stakeholder relationshipso Taken into account gender perspectiveo Empowered project managero Skilled and appropriate team members with defined roles and responsibilitieso Availability of funding

3. The role of the project managera. Empowerment

A project manager has to be empowered. With empowerment comes responsibility. The Project Manager represents the project in a professional manner and stand ups for the project when required

b. Project management skills

Page 4

o Understand knowledge, standards, and regulations applied in the projecto Understand the project environmento Understand the importance of gender perspective in development projecto General management skills (budgeting, scheduling)o Communication skillso Interpersonal skills

4. Keys to successful projectKeys to successful project is PLAN - LEAD – COMMUNICATE - MANAGEPLAN

• Identify all stakeholders • Develop the project plan before starting the project • Establish communications protocols• Define your requirements in detail – remember gender sensitive requirements• Identify risks• Make contingency plans• Plan a reasonable roll-out schedule

LEAD• Ensure strong, committed management support• Experience in project management• Establish clearly defined directions• Be proactive• Set clear performance expectations• Ask for technical assistance• Do not start roll-out until pilot is complete

COMMUNICATE• Communicate objectives frequently• Recognize different perspectives, gender perspective• Check assumptions frequently• Manage expectations• Make people aware of why gender mainstreaming is necessary• Share success and broadcast achievements• Invite feedback

MANAGEEssential management skills

• Project planning, initiation, and organization• Recruiting people, training and keeping them• Effective project negotiation• Accurate estimating and cost control• Manage unreasonable spend• Implementation and project management• Personal and project leadership• Managing risk and making decisions

Page 5

• Effective problem management• Performance management• Managing the projects within the organization• Growing and sustaining a high-performance team

A good project team can be the key to a successful project.5. Managing the contractor

o Don’t “marry” the contractoro Trust but questiono Their interest is their benefitso Your interest is a working systemo Nothing is freeo The result is only as good as your requirementso Don’t give the contractor or developer too much control, responsibility, or authority

Provide performance feedbacko Build performance reviews into the contracto Be clear and specifico Be timely

6. Managing the projectSuccessful project management is delivering good result (quality product) that meets the donor’s requirements (client) within time, scope, and budget.

o Gradual, progressive increase in the project’s scope that is not noticed immediatelyo Occurs when additional requirements result in scope change and can cause cost and

schedule overrunsKeeping control of the budget

o Monitor project cost o Determine the rational of every expenseso Change the activities of the project, reduce scopeo Avoid unapproved changes to the project

Reporting mechanism: Monthly report, Quarterly report, Annual report, Progress reportProject management reporting

o Establish reporting requirements upfronto Include the strengths, the limitations, and the mistakeso Be brief but accurateo Management really does want to knowo Including sections on Gender issue within your reports will show how gender

sensitive your project is Project Management Tips

o Project Manager does not work other than manage the projecto Project Manager spends 90% of time communicatingo Understand the project flow from beginning to endo Identify risks and develop contingencies

Page 6

o Clearly outline the critical path of the projecto Establish the right relationships o Establish a speedy conflict resolution processo Be flexibleo Be firm

Lecture 2: Result based management1. What is Results-based management (RBM)

Results Based Management (RBM) is a management strategy by which an organization ensures that its processes, products and services contribute to the achievement of desired results (outputs, outcomes & impacts).RBM rests on stakeholder participation and on clearly defined accountability for results. It also requires monitoring of progress towards results and reporting on performance/feedback which is carefully reviewed and used to further improve the design or implementation of the programme/project.RBM is a life-cycle approach to management that integrates strategy, people, resources, processes, and measurements to improve decision making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning, and adapting, as well as reporting performance.RBM is:

o defining realistic expected results based on appropriate analysis;o clearly identifying program beneficiaries and designing programs to meet their needs;o monitoring progress toward results and resources consumed with the use of

appropriate indicators;o identifying and managing risk while bearing in mind the expected results and

necessary resources;o increasing knowledge by learning lessons and integrating them into decisions; ando Reporting on the results achieved and resources involved.

Why RBM?

o Traditional management: focused inputs (what have spent), activities (what have done), and outputs (what have been produced), it did not tell the progress toward solving the problems àthe problems remained when projects were completed

o Modern management: requires looking beyond activities and outputs to focus on actual results: the changes created, and contributed to, by our project/programme.

à Establishing clearly defined expected results;

à Collecting information to assess progress regularly;

à Taking timely corrective action;

à Can manage projects or investments to maximize achievement of development results: a sustained improvement.

Pillars in RBM

Page 7

o RBM is based on four main pillars:

(i) the definition of strategic goals which provide a focus for action;

(ii) the specification of expected results which contribute to these goals and align programmes, processes and resources behind them;

(iii) on-going monitoring and assessment of performance, integrating lessons learnt into future planning;

(iv)improved accountability, based on continuous feedback to improve performance

2. The purpose and benefits of RBMo It supports achievement of positive results and objectives;o It facilitates the identification of negative results and risks and in this way allows

introducing measures to correct them;o It clarifies tasks and responsibilities and establishes feedback and discourse

mechanisms between all stakeholders;o It creates a well structured and transparent basis for decisions, based on real data and

information;o It facilitates communication about achieved results and therefore:

- Improves communication with the contractor;

- Facilitates co-financing

o It facilitates understanding and presentation of contributions to the Millennium Development Goals

3. RBM involves three major steps

Step 1: Performance planning (e.g. project design or preparation of the annual work plan and budget): use Project Logframe; Cost Tables; Results-Oriented Annual Work PlanStep 2: Performance measurement: use M&E Tools

Step 3: Performance evaluation and reporting: use Supervision reports; Annual Progress Reports; Mid-Term and Completion review reports.

Note: Project management need to regularly collect, analyze and evaluate information on results in order to make sure that expected objectives, at all levels, will be achieved.

RBM tools and processes

1. Implementation planning: using project log-frame, expense, working schedule based on annual outcome

2. Measurement/ assessment of effectiveness: using M&E tools

3. Activities assessment: using monitoring report, annual progress report, end and mid-term result report

Page 8

Input Activities Output Outcomes Impact

Report and M& E system Annual Outcome Survey Impact evaluation research

4. The Results’ Chain

a. Results’ Chain is the core of RBM, it consists of series of results – IFAD’s approach:

- Result level 1 – outputs, - Result level 2 – outcomes, and - Result level 3 – impact

Table 3: Result Chain

Results’ level Definition Example Level 3 (Impact)

Positive and negative, long-term effects produced by a development intervention, directly or indirectly, intended or unintended

Positive impact: increased farmers’ income

Level 2 (Outcomes)

The likely (or achieved) short-term and medium-term effects of an intervention

Increased capacity by small farmers to grow high-value crops; Increased production of high value crops in project area

Level 1 (Outputs)

The products and services produced (through activities) by the project and provided to direct beneficiaries

Activity of project: organized training course à result: Farmers trained in the production of high-value crops

Input Financial, material and human resources used for implementing activities

Human, material, technical

Process of achieving results The transformation of inputs on the first level result by series of activities seeming

automatically Automatic Input first- level result Example: if the project has human resource, materials and training, farmers will be trained

The first level result can’t transform automatically in second and third level results Not-automaticFirst- level result second and third level results

External agents Example: The first level result is farmer trained, but this is not sure that they have obtained high yield, because nature disaster, epidemics etc. can affect the results.

Figure 1: Project process

Page 9

1st Level results

(Outputs)

2nd Level results

(Outcomes)

Impact

RIMS 1st level indicators RIMS 2nd level indicators RIMS 3rd level indicators

Figure 2: Project Results-Chain

Table 4: Example of Result chain- IFAD’s approach

Objective hierarchy Project Logic

Goal(Impact)

Increase income from agricultural productionàfood security à Eliminate hunger, reduce poverty

Purpose(Outcomes)

70% of participants applied the techniques learned in the training

Outputs 700 famers attended the training course

Activities(Process)

Conduct 5 training courses on agricultural techniques

Inputs Staffs, Resources

b. Results’ Chain – CIDA’s approach

o Outputs

o Immediate outcomes (a change in access, ability, or skills)

Page 10

o Intermediate outcomes (a change in behavior or practice)

o Ultimate outcome (change in state)

Table 5: Example of Result chain- CIDA’s approach

Ultimate outcome

Improved health among people living in region Y of country X

Intermediate outcomes

Increased use of clean drinking water by people living in region Y

Increased use of health services by people living in region Y

Immediate outcomes

Increased access to clean drinking water for people living in region Y

Increased ability to maintain wells among people living in region Y

Increased access to health services for people living in region Y

Outputs Wells built in region Y

Training on well maintenance developed and delivered to people living in region Y

Regional health centers in region Y rehabilitated and staffed

Activities Build wells in region Y

Develop and deliver training on well maintenance to people living in region Y

Rehabilitate and staff regional health centers in region Y

Page 11

Lecture 3: Basic M&E concepts 1. Monitoring

- Monitoring is a continuous management function.- Aims primarily at providing programme managers and key stakeholders with regular

feedback and early indications of progress or lack thereof in the achievement of intended results.

- Monitoring tracks the actual performance against what was planned or expected according to pre-determined standards.

- It generally involves collecting and analyzing data on programme processes and results and recommending corrective measures

2. Evaluation

- Evaluation is a time-bound exercise that attempts to assess systematically and objectively the relevance, performance and success, or the lack thereof, of ongoing and completed programmes

- Evaluation is undertaken selectively to:

(i) Answer specific questions to guide decision-makers and/or programme managers

(ii)Provide information on whether underlying theories and assumptions used in programme development were valid, what worked and what did not work and why

- Evaluation commonly aims to determine the relevance, validity of design, efficiency, effectiveness, impact and sustainability of a programme

The Relationship between Monitoring and EvaluationMonitoring and evaluation are intimately related:

Both are necessary management tools to inform decision-making and demonstrate accountability.

Evaluation is not a substitute for monitoring nor is monitoring a substitute for evaluation.

Both use the same steps however; they produce different kinds of information. Systematically generated monitoring data is essential for successful evaluations

Table 6: Monitoring and evaluation

Monitoring Evaluation

Continuous Periodic: at important milestones such as the mid-term of programme implementation; at the end or a substantial period after programme conclusion.

Keeps track; oversight; analyses and documents progress.

In-depth analysis; Compares planned with actual achievements.

Focuses on inputs, activities, outputs, implementation processes, continued relevance, likely results at outcome

Focuses on outputs in relation to inputs; results in relation to cost; processes used to achieve results; overall relevance; impact; and sustainability.

Page 12

level.Answers what activities were implemented and results achieved.

Answers why and how results were achieved. Contributes to building theories and models for change.

Alerts managers to problems and provides options for corrective actions.

Provides managers with strategy and policy options.

Self-assessment by programme managers, supervisors, community stakeholders, and donors.

Internal and/or external analysis by programme managers, supervisors, community stakeholders, donors, and/or external evaluators.

Box 1: Five key evaluation questions

1. Relevance – Was/Is the project a good idea given the situation needing improvement? Does it deal with target group priorities? Why or why not?

2. Effectiveness – Have the planned purpose and component objectives, outputs, and activities been achieved? Why or why not? Is the intervention logic correct? Why or why not?

3. Efficiency – were inputs (resources and time) used in the best possible way to achieve outcomes? Why or why not? What could we do differently to improve implementation, thereby maximizing impact, at an acceptable and sustainable cost?

4. Impact – To what extent has the project contributed towards its longer-term goals? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise? To what extent has the project contributed towards poverty reduction (or other long-term goals)? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise?

5. Sustainability – Will there be continued positive impacts as a result of the project once it has finished? Why or why not?

3. Important terms

- Goal: The higher-order programme or sector objective to which a development intervention, such as a project, is intended to contribute. Thus it is a statement of intent.

- Objective/Purpose: A specific statement detailing the desired accomplishments or outcomes of a project at different levels (short to long term). A good objective meets the criteria of being impact oriented, measurable, time limited, specific and practical. Objectives can be arranged in a hierarchy of two or more levels 

- Indicators: Quantitative or qualitative factor or variable that provides a simple and reliable basis for assessing achievement, change or performance. A unit of information measured over time that can help show changes in a specific condition. A given goal or objective can have multiple indicators.

- Target: A specified objective that indicates the number, timing and location of that which is to be realized.

Page 13

- Result: The measurable output, outcome or impact (intended or unintended, positive or

negative) of a development intervention.

- Input: The financial, human and material resources necessary to produce the intended outputs of a project.

- Output: The tangible (easily measurable, practical), immediate and intended results to be produced through sound management of the agreed inputs. Examples of outputs include goods, services or infrastructure produced by a project and meant to help realize its purpose. These may also include changes, resulting from the intervention, that are needed to achieve the outcomes at the purpose level.

- Outcome: The results achieved at the level of "purpose" in the objective hierarchy. In IFAD's terminology, outcome is part of impact (result at purpose and goal level).

- Impact: The changes in the lives of rural people, as perceived by them and their partners at the time of evaluation, plus sustainability-enhancing change in their environment to which the project has contributed. Changes can be positive or negative, intended or unintended. In the logframe terminology these "perceived changes in the lives of the people" may correspond either to the purpose level or to the goal level of a project intervention.

- M&E system: The set of planning, information gathering and synthesis, and reflection and reporting processes, along with the necessary supporting conditions and capacities required for the M&E outputs to make a valuable contribution to project decision-making and learning.

- M&E framework: An overview of the M&E system developed during the design phase of a project and included in the project appraisal report.

- M&E matrix: A table describing the performance questions, information gathering requirements (including indicators), reflection and review events with stakeholders, and resources and activities required to implement a functional M&E system. This matrix lists how data will be collected, when, by whom and where.

- M&E plan: An overall framework of performance and learning questions, information gathering requirements (including indicators), reflection and review events with stakeholders, and resources and activities required to implement a functional M&E system

- Supervision: A process in which the legally responsible organization (cooperating institution or IFAD itself) administers the loan, periodically reviews progress towards objectives, identifies key obstacles, helps find workable solutions and makes strategic changes, as required.

- Monitoring: The regular collection and analysis of information to assist timely decision making, ensure accountability and provide the basis for evaluation and learning. It is a continuing function that uses methodical collection of data to provide management and

Page 14

the main stakeholders of an ongoing project or programme with early indications of progress and achievement of objectives.

- Evaluation: A systematic (and as objective as possible) examination of a planned, ongoing or completed project. It aims to answer specific management questions and to judge the overall value of an endeavour and supply lessons learned to improve future actions, planning and decision-making. Evaluations commonly seek to determine the efficiency, effectiveness, impact, sustainability and the relevance of the project or organization’s objectives. An evaluation should provide information that is credible and useful, offering concrete lessons learned to help partners and funding agencies make decisions.

- Monitoring and evaluation (M&E): The combination of monitoring and evaluation which together provide the knowledge required for: a) effective project management and b) reporting and accountability responsibilities.

- Control group: A specially selected subgroup of people who purposefully do not receive the same treatment, input or training, etc. as the target group. Thus, differences between the control group and the target group can be measured and evaluated.

- Gender Mainstreaming: Gender mainstreaming is a strategy for developing policies, legislation, programs, project, society and community’s development plans from gender equality perspective in order to benefit both women and men.

Page 15

Lecture 4: M&E system1. What is M&E system?

The M&E system is the set of planning, information gathering and synthesis, reflection and reporting processes, along with the necessary supporting conditions and capacities required for the outputs of M&E to make a valuable contribution to decision-making and learningKey project stakeholders need to develop the different elements of the system together if they are all to use the outputs to improve implementation.The M&E system consists of four interlinked parts:

(i) Setting up the M&E system(ii) Implementing the M&E system means gathering and managing information(iii) Involve project stakeholders in reflecting critical(iv) Communicating and reporting the results of M&E

The results from M&E – both the communication processes and the information will improve the project strategy and operations. So for M&E to work as a tool for managing for impact, project management and the M&E staff need to be clear on how to identify, agree upon and follow up on project improvements. If this process is not done properly, even a good M & E system is not contributing much better value for the project.

Table 7: M&E tasks

Early design phase (formulation and

appraisal)

Establish the scope and purpose of the M&E system

Indicate key performance questions and indicators, plus associated monitoring mechanisms Identify organizational arrangements for M&E

Develop terms of reference for M&E staff Indicate the process for how M&E is to be established during start-up

Start-up prior to loan effectiveness (with the Special Operating Fund)

Revise performance questions, indicators and monitoring mechanisms after reviewing the project strategy

Organize training with staff and partners likely to be involved in M&E

Initiate a project implementation manual with key staff

Prepare a project implementation manual with key staff

Start-up after loan effectiveness

Review project design in relation to M&E with key stakeholders

Develop a detailed M&E plan, taking into consideration existing mechanisms with partners Put in place necessary conditions and capacities for M&E to be implemented

Main implementation

Ensure information needs for management are met

Coordinate information gathering and management Facilitate informal information gathering and communication

Support regular review meetings and processes with all implementers

Page 16

Prepare for supervision missions Prepare for and facilitate the annual project review

Conduct focused studies on emerging questions Communicate results to stakeholders Prepare annual progress reports

Mid-term review

Collate information for the mid-term review Facilitate the internal review process to prepare for the external review process Help respond to MTR feedback Adjust the M&E system. As necessary

Phasing-out and completion

Assess what the implementers can do to sustain impact and sustain M&E after closing down-and implement these ideas Hold workshops and do field studies with key stakeholders to assess impacts Identify lessons learned for the next phase and/or other projects

2. How to set up the M&E system Six steps to set up the M&E system:

Step 1: Establishing the purpose and scope – Why do we need M&E and how comprehensive should our M&E system be.When formulating the purpose at appraisal or revisiting it during start-up, ask yourself the following question: What are the main reasons for setting up and implementing M&E, for us – as

implementing partners and primary stakeholders – and for other key stakeholders?Define the scope of the M&E system by asking: What level of funding is potentially available? What level of participation in M&E by primary stakeholders and partner

organizations is desirable and feasible? How detailed does the M&E information have to be, either in terms of quantitative or

qualitative data? What sort of baseline study is desirable and feasible? What are the current M&E capacities among primary stakeholders and partners

organizations, and how will this affect the desired levels of M&E?Step 2: Identifying performance questions, information needs and indicators - What do we need to know to monitor and evaluate the project in order to manage it well? A performance question such as: if the project is performing as planned or if not, why not Working with performance questions to guide indicator analysis will give you more

integrated and meaningful picture of overall project achievements Answering these questions requires descriptive analysis and quantitative information Starting by identifying performance question make it easier to recognize which

specific indicators are really necessary

Page 17

Sometimes a performance question can be answered directly with a simple

quantitative indicator. However, very often the question can only be answered with a range of qualitative

and quantitative information.Box 2: The basic performance questions per level of the objective hierarchy

Activities – What have we actually done?

Outputs – What have we delivered as a result of project activities (e.g., number of people trained)?

Outcomes (results) – What has been achieved as a result of the outputs (e.g., extent to which those trained are effectively using new skills)?

Impacts – What has been achieved as a result of the outcomes (e.g., to what extent are NGOs more effective)? What contribution is being made to the goal? Are there any unanticipated positive or negative impacts?

Lessons – What has been learned from the project that can contribute to improved project implementation or to building relevant fields of knowledge

Box 3: Examples of performance questions

Performance questions are not just about what has been achieved. They also ask why there is success or failure and what has been learned to improve future action. Examples of performance questions include:

How has the purchasing power of target households as compared to non-target groups? What external factors have influenced any changes?

To what extent are target households better able to meet their housing, education and health needs than non-target households?

How have the diversity, production and productivity of agriculture in the target area changed as a result of project activities and as a result of external factors?

Step 3: Planning information gathering and organizing – How will the required information be gathered and organized? For each information need or indicator, you must establish how the information will

be collected and organized. Not only will each indicator require choosing a different method, but for each

indicator or information need you will usually present several options. There are many methods – qualitative and quantitative and individual versus group-

based. Particularly critical at the moment of method selection is to determinate who will be

involved in collecting, compiling and analyzing.

Page 18

The more that the intended users of the methods can involve in selecting or

developing the methods, the more chance there is that they will understand them and use them correctly.

If methods are selected by someone not using them, then training users in the method will be essential.

Gathering data is one thing. But each bit of information also needs to be collated, perhaps summarized and certainly analyzed by the right people.

Step 4: Planning critical reflection processes and events – How will we make sense of the information gathered and use it to make improvements? Critical reflection can occur formally and informally. Formally, it can be facilitated during project meetings, workshops with partners and

primary stakeholders or as part of external evaluation. Informally, it can occur in ongoing discussions between project stakeholders.

Table 8: Critical Reflection Schedule for a project

Critical Reflection Processes and

Events

Purpose and Description Whom to involve Timing

Participatory review of project strategy

Update the situation analysis, revise problems/visions, adjust objective hierarchy and assumptions

Representatives of intended primary stakeholders, staff or participant organization, all project staff, facilitator

Three-day workshop at start-up

Development of M&E plan with stakeholders

Assess different information needs, take stock of who is already doing what, agree on priority information areas, refine questions/indicators, decide on methods, agree on responsibilities

Representatives of intended primary stakeholders, staff or participant organization, all project staff, facilitator

Four or five full-day meetings during the first six months of the project

Quarterly progress reviews by PMU staff

Discussion of key successes and problems

Senior staff of PMU and partner organization

One-day meeting every three months

Field visits Firsthand look at what is happening in the field, informal chats about how activities are being implemented

Field staff, supervisors of field staff, project director

Weekly visit for field staff, monthly for the project director

Annual project review

Summary of key successes and problems, ideas for changing project

Representatives of intended primary stakeholders, staff of

Once a year

Page 19

activities/outputs and assumptions, review of implications for the project logframe, identification of lessons learned about project implementation, M&E system adjustment

implementing partners, all project staff, facilitator

Periodic review workshops of key project components

Focused discussion about the strategy and operations of key components to adjust the objective hierarchy, solve problems and identify lessons learned

Key stakeholders of the project component: intended primary stakeholders, implementing partners, field and senior project staff

Once a year in the first two years; after that, once every two years

Preparation for supervision missions

Explain the mission purpose, agree on what the project and stakeholders would like to get out of the mission, identify who needs to prepare what before the mission, organize the logistics

Small group of primary stakeholders representatives, senior staff of the participant organization, senior project staff

One month prior the supervision mission

Step 5: Planning for quality communication and reporting – What, how and to whom do we want to communicate in terms of our project activities and processes?

When reporting on progress with the AWPB, you will direct yourself to funding agencies, steering committees, cooperating institutions and implementing partners.

Primary stakeholders have a right to knowing overall how the project is progressing and they deserve the opportunity to react to initial findings.

Funding agencies and managers need information on impact, while all implementing partners need to understand problems in order to find solutions

Two sets of M&E findings will need to be communicated◦ First, it is good practice to discuss draft M&E findings with implementing

partners and primary stakeholders in order to get feedback on accuracy, reach joint conclusions and agree on next steps.

◦ This second set of final findings will fulfill accountability needs but can also serve for advocacy purposes.

Step 6: Planning for the necessary conditions and capacities – what is needed to ensure that the M&E system actually works?Table 9: Questions to guide the detailed planning of M&E conditions and capacities

Conditions and Capacities

Questions to Guide Detailed Planning

Human capacity for M&E

- What are the existing M&E capacities?- What training will be necessary?

Page 20

- What consultancy support will be required?

Incentives for implementing M&E

- Are M&E responsibilities included in job descriptions and terms of reference?

- How will reflection and learning among staff, partners and the intended primary stakeholders be encouraged?

Organizational structures

- Will there be an M&E unit or will M&E be spead among all parties? If there is a unit, how many people will it have and where will it be located, under whose authority?

- How closely connected will M&E staff be with project management?Management information systems (MIS)

- What information must be stored and accessible, when, how and for whom?

- What level of computerization is required and appropriate?- What expertise will be required to set up the information management

system?Financial resources - Is there a separate M&E budget and have sufficient resources been

allocated?- Has the staffing allocation for the project taken into account time for all

relevant staff to undertake M&E activities?3. M&E Plan3.1 The structure of the monitoring planThe monitoring plan specifies 5 issues: (i) What are the monitored subjects?; (ii) When is the monitoring conducted?; (iii) How is the monitoring conducted?; (iv) Who carry out the monitoring?; and (v) Monitoring cost.

(i) What are the monitored subjects?Important factors in determining the monitored subjects, including:

- Main subjects that receive information;- Questions;- Indicators (for changes);- Information on the implementation;- Unexpected changes;- General problems;- Initial evaluation on the effectiveness of the results.

(ii) When the monitoring is conductedAfter identifying the monitoring objectives, indentify the timeline for the monitoring. It is necessary to determine a timeline suitable with the timeframe of the whole project.

(iii) How the monitoring is conductedGroup discussion to determine the monitoring method, example:

Monitored issues

IndicatorsHow to measure the

indicatorsHow to report the

resultsBuild models for crop production

Number of participants (households)

Report of project officials on the

Were the objectives achieved?

Page 21

for project households

Number of households that apply the model

number of participantsQuestionnaires

Adjustment measures (if objectives were not achieved)?

(iv)Who conduct the monitoring? In the monitoring plan, it is necessary to present the name and position of the monitoring official. This presentation has two effects, (i) link the monitoring official with specific activities and hold him/her responsible for his/her activities; and (ii) the monitoring official knows what, how and when to do.

(v) Monitoring cost Whether there are expenditures for the monitoring. If there are, estimate the monitoring cost, via monitoring activities and supports, such as means, for best results.3.2 The structure of the evaluation planAn evaluation plan specifies 5 issues: (i) The evaluation objectives; (ii) Evaluation methods; (iii) Organization of the evaluation; (iv) Evaluation cost; and (v) Evaluation information-use plan.

Table 10: The structure of an evaluation planRequirement Content

Objectives, evaluation content

What are evaluated? – What we need to know? Base on the project design – the Objectives section; or the Project Evaluation Plan – Evaluation Objectives section; or the Term of Reference to identify the evaluation objectives and content.

Evaluation method Present the evaluation methods used to collect information in order to specify the evaluation objectives and content.

Organize the evaluation

Who will do what? What is his/her role? What are the roles of the partners, the relevant stakeholders and the evaluation team?

Evaluation cost Full estimated expenditure for each evaluation activity.Evaluation information-use plan

Structure of the evaluation report.Expected purposes of information.Plan to use and share information.

The five standard types of evaluation questions1. Relevance – Was/Is the project a good idea given the situation needing improvement? Does it deal with target group priorities? Why or why not?2. Effectiveness – Have the planned purpose and component objectives, outputs, and activities been achieved? Why or why not? Is the intervention logic correct? Why or why not?3. Efficiency – were inputs (resources and time) used in the best possible way to achieve outcomes? Why or why not? What could we do differently to improve implementation, thereby maximizing impact, at an acceptable and sustainable cost?4. Impact – To what extent has the project contributed towards its longer-term goals? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise? To what extent has the project contributed towards poverty reduction (or other

Page 22

long-term goals)? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise?5. Sustainability – Will there be continued positive impacts as a result of the project once it has finished? Why or why not?3.3 M&E and project logic frameThe log frame is a tool to present objectives into specific and measureable indicators. In addition, the log frame specifies how to address those objectives as well as assumptions to achieve them.This tool also clarifies the relation between the objectives of the project and the activities in the project design. A specific log frame can be described as follows:o Long-term objectives: impacts, purposes, expected long-term results – contributions of the

project.o Medium-term objectives: results, purposes of the projects, intermediate results – short-

term and medium-term benefits that the project deals with.o Short-term objectives: quick outputs and results – products or services that are directly

provided by the project.o Indicators: To monitor and evaluate the results of a project/programme, measure

impacts, results and outputs as expected to be built. These measuring methods are usually called indicators.

o Criteria: Once the project objectives and relevant indicators are built, project management officials and M&E officials need to identify annual project evaluation method or criteria on the basis of final expected project’s results

- If the progress criteria, final output criteria and result criteria have been identified in the log frame in the drafting of the project’s plan, M&E officials still have to evaluate their appropriateness in the operational environment.

- In case the final output criteria are agreed by the sponsor but the annual progress criteria have not been established, the project officials and M&E officials have to use completion criteria of the project to determine suitable annual criteria. Practical criteria can be built by dividing indicators into different levels of completion.

o Risk: All projects face a certain kind of risks that hamper the achievement of objectives and/or expected results. Thus, these risks need to be identified, monitored and evaluated closely by M&E officials to propose necessary responses to reduce the impacts of any internal or external risk.◦ Internal risk:

delays in signing contract that affect important decisions delays in disbursement for stakeholders’ operation change in number of staff, etc

◦ External risk: external risks lie outside the project but still affect its operational environment. Thus, they are often out of the control of the project. However, they still need to be monitored, evaluated and controlled right after they are recognized as potential threats to the project.

Page 23

o Risk management and risk planning: need to be built, especially for projects in unstable

environment or potentially unstable environment. In those projects, specific actions should be planned in preparation for worse conditions.

o Assumption: External factors (e.g. events, conditions or decisions) may affect the progress or the success of a project/programme. The initial assumptions are critical requirements for the success of a project/programme. Risk assumptions are conditions that may threaten the implantation of a project/programme.

3.4 Detailing M&E planDetailing M&E plan based on project appraisal reportA good appraisal report will include an indicative M&E framework that provides enough detail about these questions to enable budgeting and allocation of technical expertise, give funding agencies an overview of how M&E will be undertaken, and guide project and partner staff during start-up. But this will only be indicative and needs to be adjusted and detailed further during the start-up phase

Table 11: Task needed when detailing the M&E plan based on a project appraisal report

M&E Design Steps

Outputs in Project Appraisal Report (the M&E

Framework)

Task during Project Start-up to Develop a Detailed M&E Plan

1. Establishing the purpose and scope

Broadly defined purpose and scope of M&E in the project context

Review the purpose and scope with key stakeholders.

2. Identifying performance question, indicator and information needs.

List of indicative key questions and indicators for the goal, purpose and output levels

Assess the information needs and interests of all key stakeholders.

Precisely define all questions, indicator and information needs for all levels of the objective hierarchy.

Check each bit of information for relevance and end-use.

3. Planning information gathering and organizing

Generally described information gathering and organizing methods to enable resource allocation

Plan information gathering and organizing in detail (who will do use which method to gather/synthesize what information, how often and when, where, with whom, with what expected information product).

Check the technical and resource feasibility of information needs, indicators and methods.

Develop formats for data collection and synthesis

Page 24

4. Planning for quality communication and reporting

Broad description of key audiences and types of information that should be communicated to them to enable resource allocation

Make a precise list of all the audiences, what information they need, when they need it and in which format.

Define what is to be done with the information – simply send it, provide a discussion for analysis, seek relevant feedback for verification, etc.

Make a comprehensive schedule for information production, showing who is to do what by when in order to have the information ready on time.

5. Planning critical reflection processes and events

General outline of key processes and events

Precisely detail which methods/approaches are to be used, with which stakeholder groups and for what purpose.

Identify who is responsible for which reflective events

6. Planning for the necessary conditions and capacities

Indicatives staffing levels and types, clear description of organizational structure of M&E, indicative budget

Come to a precise definition of: the number of M&E staff, their responsibilities and their linkages, incentive needed to make M&E work, organizational relationships between key M&E stakeholders, the type of information management system to be established and a detailed budget.

3.5 Using the M&E Matrix for detailed planningStep 1: Identifying performance questions- Identifying performance questions helps you focus your information gathering on what

you will really use for understanding and improving project performance.- Identifying performance questions (indicators and selection methods) will be iterative:

make an initial choice, assess its feasibility, accept and use it or reject it and find the next option.

Step 2: Identifying information needs and indicators- Using your performance questions, you can more easily identify useful indicators and

other information need for which you will need to collect data. Only data that help answer your performance questions are necessary.

- This helps avoid collecting information that is difficult to use to guide the project strategy and operations.

Step 3: Knowing what baseline information you need- Many baseline studies suffer from information overload and lack of use.When deciding whether you need to collect baseline data for a particular performance question, ask yourself if you need to compare information to be able to answer the question. If not, or if information already exists, then you will not need to collect baseline data

Step 4: Selecting which data collection methods to use, by whom and how often

Page 25

- Once you have decided what information is needed and what indicators will be use, you

need to decide which methods will be used for gathering the data.- You have many options: methods that are more qualitative or more quantitative, more

or less participatory, and more or less resource intensive.Each will provide information of varying degrees of accuracy and reliability

Example:To gather information on food security changes you could consider three different methods:(1) A detailed household survey conducted by independent researchers,(2) A participatory assessment process where women household members do their own monitoring and discuss their findings, or(3) Focus-group meetings to discuss changes that specific social groups have experienced. Before embarking on resource intensive data-collection exercises, carefully consider whether a simpler method would yield sufficient information of good enough quality for your purpose.Step 5: Identifying the necessary practical support for information gathering- For a method to lead to the information you require, you will need to organize the

conditions to make it work. - These are often forgotten in the focus on identification of indicators but are critical to

success.For each method, consider if and how you need to:

o Develop forms to record data;o Develop forms, filling systems and databases for collating and storing information;o Train staff, partners or community members who will be involved;o Check and validate data; o Organize external M&E or research expertise that may be needed;o Agree on responsibilities for different tasks;o Ensure everyone has sufficient financial resources and equipment.

Step 6: Organizing analysis, feedback and change- In the rush to get out and start collecting data, many M&E units pay insufficient attention

to the process of using the information for analysis and directing changes in the project.- To make sure that data will be used – and not just collected – think about how you will

organize the analysis of information for each performance question. - Sometimes a performance question cannot be answered without prior analysis of several

bits of information. Who will do it? When will it happen? Also consider what form information should be in so that it can be used by different stakeholders: present information visually, in graphs or maps or meetings to get more feedback.

3.6 Indicative contents for an M&E Plan- The operational plan for project M&E provides the conceptual and, above all, practical

basis for planning, monitoring and evaluation within the project.- The document also defined the main concepts related to M&E, the redesigned logframe

matrix, the adjustment of information collection systems and the database of primary stakeholders.

Page 26

- The M&E operational plan will be the reference point for stakeholders throughout the

project life. So it needs to be comprehensive enough, at macro and micro level.Table 12: Indicative contents for an M&E operational Plan

Topic Description Purpose and scope Project overview and objectives, rationale and justification for

the design of the M&E system How the M&E system will support project management and meet

the reporting requirements and information needs of different stakeholders.

Summary of overall experience of M&E undertaken with key stakeholders

Discussion of extent of participation, balance between qualitative/quantitative approach, resource intensiveness and the intended poverty focus of the M&E system

Approach Overview of how stakeholders will be involved, what learning-oriented approaches will be used and, in general terms, what information gathering and analyzing methods will be used; for example, the extent of use of participatory approaches, geographic information systems, computer-based information systems or baselines surveys

Revised logical framework, plus performance questions, indicators, information needs and sources

Precise definition of all performance questions, indicators and information needs for all levels of the objective hierarchy:

Assessment of the information needs and interests of all key stakeholders

Assessment of the indicators of exogenous factors and assumptions (e.g., climate, prices, outbreak of pests and disease, economic situation, policy environment)

Assessment of the information needs and indicators for relevance and end-use and for technical and resource feasibility

Selection of performance questions and indicators Management information system and reporting

Purpose of the management information system Organization of information gathering and synthesis:

- For each expected information product – who, what, when and where

- Schedule of information production – who what, when, to whom, for what purpose

- How computerized networks and manual archiving systems are expected to function, with/to whom, for which data

- Outline of data storage needs Expected reporting outputs, for example:

- Informal communication and feedback channels- Report flows – deadlines and frequencies

Page 27

- AWPBs – outline of the AWPB format, including

output/activity plans and budgets, consolidated budgets, a training plan, a procurement plan, a contracted services plan

- Annual progress reports for the project as a whole and each component, village-based reviews

- Annual financial reports- Recurrent supervision missions

The M&E work plan (processes and events) and timing of activities

Precise definition of methods to be used with different stakeholder groups for two core purposes:

1. M&E resources, activities and implementation for effective project operations: Project resources: transport use, allowances, register for

assets, register of services/technical assistance Project activities: training (workshops, study tours, etc.),

construction (technical or social infrastructure), scheme organization, trials and demonstrations, credit lines, etc.

Other monitoring activities2. M&E of outcomes and impact for guiding the project

strategy Proposed surveys: baseline/household, component, staff Participatory annual assessment and planning workshops Other annual evaluation and beneficiary assessments, reviews,

and planning sessions Midterm review and project completion report Feasibility of methods in terms of technology and resources M&E work plan schedule: integrated schedule of key events

and reporting/decision making moments Critical event agenda

Establishing conditions and capacities

M&E organization: Necessary institutional and stakeholder linkages for M&E Existence (or not) of a specific M&E unit and how it relates to

the project structure and hierarchy of authority Human resource needs: Number, capacities and responsibilities of different

stakeholders in M&E, including project staff and primary stakeholders

Training needs of stakeholders and staff Resource needs: Vehicles and equipment Technical assistance

The M&E budget Detailed budget allocation Appendices Original and revised logframes

Page 28

List of proposed indicators Outline formats for data collection, annual and biannual

schedule of activities, etc. Outline formats for preparing: quarterly, biannual and annual

reports, a summary of main project achievements; status reports on project inputs and resources, project outputs and results, evaluation studies – summary of findings and recommendations.

Baseline survey questionnaire Staff job descriptions and details of allowances Technical Assistance terms of reference M&E work plan Detailed budget of M&E

4. Data collection methodsThere are many methods commonly used for M&E. They have been grouped as follows:

o Sampling methods;o Core M&E methods (such as stakeholders analysis and questionnaires);o Discussion methods for groups (such as brainstorming and role plays);o Methods for spatially-distributed information (such as maps and transects);o Methods for time-based patterns of change (such as diaries and photographs);o Methods for analyzing relationships and linkages (such as impact flow diagrams and

problem trees);o Methods for ranking and prioritizing (such as matrices).

Before selecting your methods, first consider three important aspects:(i) Quantitative versus qualitative methods;(ii) Individual versus group-based methods;(iii) The extent to which a method can be participatory.

Table 13: Examples of multi-purpose M&E methods

Qualitative data Quantitative data Methods for groups

Case studies, brainstorming, focus groups, SWOT, drama and role plays, maps, transects, GIS, historical trends/timelines, seasonal calendars, rich pictures, visioning, flow diagrams, well-being ranking

Nominal group technique, maps, transects, historical trends/timelines, seasonal calendars, flow diagrams, matrix scoring and ranking

Methods for individuals

Semi-structured interviews, case studies, maps, transects, diaries, historical trends/timelines, seasonal calendars, flow diagrams

Biophysical measurements, structured questionnaires, maps, transects, GIS, diaries, flow diagrams

Quantitative and qualitative methods Quantitative methods

Quantitative methods are research techniques that are used to gather quantitative data — information dealing with numbers and anything that is measurable. Quantitative methods answer questions such as: What? How much? How many? When?

Page 29

These methods are useful to provide numerical and quantified results that can be aggregated.

Qualitative methods Qualitative methods aim to acquire an in-depth understanding of human behaviors and the reasons that govern such behaviours. Qualitative methods are used to investigate the why and how, not just the what, where, when.

Table 14: Comparison of quantitative and qualitative methods

Quantitative Methods Qualitative Methods

Use To numerically measure “who, what, when, where, how much, how many, how often”.

To qualitatively analyze “how and why”.

Examples Standardized interviews; surveys using closed-ended questions; observation.

Free and guided interviews (including focus group); surveys using open-ended questions; observation; interpretation of documents.

Strengths - Provide quantitative, accurate and precise “hard data” to prove that certain problems exist ;- Can test statistical relationships between a problem and apparent causes;- Can provide a broad view of a whole population;- Enable comparisons;- Establish baseline information which can be used for evaluating impact.

- Useful when planning a programme concerned with social change;- Provide a thorough understanding of programme/project context in order to interpret quantitative data;- Provide insights into attitudes, beliefs, motives and behaviors of a small sample population (families, communities);- Establish baseline information which can be used for evaluating qualitative outcomes (changes in knowledge, attitudes, behaviors, institutional processes etc.);- Useful in case of money and time constraints;- Useful for getting feed-back from stakeholders.

Limitations

- Exactly, but it cannot measure plans, opinions, views- Unable to explain the underlying causes of the problem- Failure to provide the information about context- Expensive (large sample)

- Often not representative, does not allow generalization - Easily influenced by the opinions of interviewer and interviewee - Information collected is difficult to synthesize, analyze

Common tools to collect data and information for IFAD projectso Compilation of data from project staff/agency records or service providers’ records o Compilation of data from beneficiaries’ records

Page 30

o Survey: (i) Baseline survey; (ii) Annual Outcome Survey; (iii) End-line surveyo Key informant interviewo Focus group discussiono Observation method o Annual supervision mission

5. Gender and M&Ea) The importance of gender in M&EIf M&E does not capture such gender differences, it could lead to following

problems:◦ Can have negative impact on women;◦ Not be able to achieve the desired outcome because we are not addressing the

constraints of women and men appropriately;◦ Can make either men or women unsatisfied with the project, since they feel that their

needs are ignored;◦ Might overlook gender differences in vulnerability and might underestimate the

impact of external pressure on women and men;◦ Can lose the opportunity to empower women (improve their confidence, self esteem,

build capacity of leadership and self-organization) because we are not able to identify the trends that might be leading to their empowerment

Integrating gender into M&E system will help:◦ To measure the extent to which a project has addressed the different needs of men

and women, and has made an impact on their lives and overall social and economic well-being

◦ To improve project performance during implementation, allow for midterm corrections, and make it possible to derive lessons for future projects.

b) What gender sensitive M&E system will monitor?Gender-sensitive M&E will monitor the project’s impact on gender relations and women’s empowerment. Some aspects of gender should be focused including:

Gender division of laboro Who is more responsible for working in the household, women or men?o When the project has been implemented, has male shared the job with female or

female has to work more?Gender differences in access and control over resources (eg. Income, employment, land, social services)o Who controls income in the household?o Who participated in job training more, female or male? And have the gender

difference in applying the knowledge to household economics.o Who has written on the land use certificate more? And the gender difference in target

of using land?o The gender difference in receiving social service: influence into female’s health and

ability to access information.Gender differences in information and knowledge

Page 31

o Are there gender difference in accessing the same information (about amount of

information and how to access)o The difference in amount of information made opportunities different between

female and male.Decision making patterns in the household and communityo Who in the household has the decision power? (Compare with the contribution of

female and male in total income: whether person contributing the most in total income has the decision power?)

o The participation of female and male in activities of community (the voice and respectfulness opinions in community activities).

Women and men’s attitude and self-confidenceo The gender difference in awareness of project (object, target and activities), then

evaluate gender difference in participating in project activities.Women and men’s attitude and self-confidenceo The difference between female and male about self-confidence in all activities (the

participation and responsibility)Gender differences in vulnerability and coping strategyo E.g. Differences in dealing problems and in adjusting to external shock

c) How do we do gender and M&EGender and M&E should not only collect data, but:

o Analyze the reasons for these changes and gender differences;o Develop appropriate and feasible interventions;o Make concrete pathways towards implementation of the recommended intervention.

How do we do gender and M&E in evaluating impact:o Basic principle for gender and M&E is to disaggregate all data by sex;o Disaggregating by head of household is not enough, since if we interview only the

head of household (who are often men) we will miss out perspective of women à Select both women and men respondents during sampling;

o Designing gender-based questions for survey, focus group discussion and in-depth interview.

6. M&E implementation responsibilities- In each province, the PCU will have the overall responsibility for the implementation and

the coordination of the M&E activities.- An M&E Unit composed of a Senior M&E Officer and a Knowledge Management

Officer will be created within each PCU that will take the primary responsibility to meet the objectives described above.

- These staff will develop and operate the project's monitoring and evaluation system for monitoring the progress of the project. They will also coordinate evaluation activities for assessing impact.

- The Knowledge Management Officer will also assume the responsibility to identify, document, and disseminate good practices and lessons learned that emerge from the project as well as to identify, obtain, adapt and distribute relevant best practices and

Page 32

lessons learned that have been generated by other IFAD projects in the country or other organizations.

- At the District-level, an M&E Officer within the District Agri-business Support Unit will implement M&E activities at the District-level and provide support to the Commune Development boards for gathering accurate M&E information.

- In the first year of the project, the M&E staff will focus on establishing a functional M&E system for the project.

- From the second year, the M&E staff will focus on timely compilation of monitoring information for decision-makers and reporting and conducting annual surveys.

- In the third year of the project, the M&E staff will coordinate the social audit and mid-term evaluations.

Responsibility of Senior M&E Officer in the PCU- Setting up the M&E system, including developing/adapting formats and defining

processes as per the M&E framework;- Providing guidance and direction to district/commune officers and implementing

partners for effective implementation of M&E activities and regular reporting;- Compiling quarterly and annual project progress reports;- Organizing and coordinating the baseline survey, annual surveys, mid-term evaluation

survey and final impact survey;- Coordinating the communication of M&E results and information sharing with project

staff, stakeholders, and implementing partners,- Assisting DARD in developing a monitoring system including the forms and

implementation (TQP); and- Facilitating a participatory review of M&E outputs and the use of M&E information for

sound decision-making aimed at improving project efficiency and effectiveness.Responsibility of Knowledge Management Officer

- Managing M&E databases generated by the project;- Coordinating the identification and documentation of good practices and lessons learned

being generated by the project;- Ensuring that reports and other documents produced by the project are circulated to

decision-makers, including the project steering committee, in a timely fashion;- Managing a circulation list of stakeholders and other organizations and ensuring that

documents produced by the project are disseminated as appropriate to parties listed;- Responding to requests from decision-makers to obtain information on good practices

and lessons learned from other projects or organizations. Responsibility of District level M&E staff

- Collecting commune reports and aggregating them in district reports; - Providing data and information on project activities undertaken at the district level; - Supporting and assisting community facilitators in the implementation of M&E activities

at the commune level;- Assisting the PCU M&E unit in conducting annual surveys and the social audit; and

Page 33

- Collecting information from the district line agencies to measure institutional indicators.

Commune level- Commune level data will be collected by the CDB, who will also engage project

beneficiaries and other stakeholders in conducting commune-level participatory monitoring and evaluation activities and will provide support to survey teams carrying out the annual surveys, social audits and impact evaluation surveys;

- Project implementing partners, including government agencies and community based organizations, will play a crucial role in supporting the project M&E by providing relevant data on the interventions undertaken and the results achieved.

Page 34

Lecture 5: Gender mainstreaming in development projects1. Basic definition when conducting gender analyzing Sex and gender

Sex refers biological characteristic of men and women.Gender intends to emphasize the characters, position and the role of men and women in social relationships.

Table 15: Discriminate between sex and gender

Sex Gender

Characteristics

Biological characteristics Social relationships

Congenital Education, social environment

Homogeneous Diverse, there are differences between cultures

Can’t change Can change

Example: Women can give birth Example: Women work as prime minister and pilot or Men take care children

The role of genderThe role of gender is presented through activities implemented by men or women; it can be changed due to time, conditions and circumstances • Productive role: Productive work is making things or providing services that can be

consumed or traded for money.  • Reproductive role: Reproductive work is the care and maintenance of the household

and the family. It includes bearing and caring for children, cooking, collecting water and fuel, cleaning, shopping, mending and looking after the disabled, old or sick members of the family.

• Community role: Community work is the collective organizing of social events such as funerals, churches, welfare organizations, civics, community meetings as well as local political activities

Both men and women can participate in three roles. In some societies, women have to take part fully in reproductive activities (doing housework and taking care children), participate relatively more in productive activities and virtually no participation in community activities. In-depth understanding about gender roles help us design appropriate job for men and women; attract the effective participations of both men and women and reduce gender inequality in the social division of labor Gender prejudice

Gender prejudice is a bias in perceptions, attitudes about the role, position and capacity of men or women.Gender prejudice often have a incorrect view about the capacity of the individual, it shows the views of a group people or the community about what men and women should do or not do.

Page 35

Example: women should not have high education, men are breadwinners and participate in community activities Basic needs and strategic needs

Basic needs: refer to those fundamental requirements that serve as the foundation for survival. Basic needs are defined as emergency food, clothing; job and emergency housing assistance. If people meet these needs, their life is improved and it does not affect to division of labor as well as the position of women in society. Strategic needs: refer to the need relating to division of labor, power and control by gender. These needs can change due to social, economic and political conditions. When these needs are met, there are changes in division of labor. Gender equitable

As men and women have the same position and role and equal opportunity to develop their capacity for the development of the community, the family and equal enjoyment of the fruit of development (Gender equality law). It does not mean that women and men are the same. It means that the rights, responsibilities and opportunities are independent on their gender-male or female.Gender equitable is considered as target to be achieved.Gender equity is fairness in the treatment with men and women according to their respective needs. It may include same treatment or different treatment but it is considered as equivalent in terms of rights, benefits, obligations and opportunities.Equity is the way to get the gender equality.2. Gender mainstreaming in development project2.1 What is gender mainstreaming?

Gender mainstreaming is a strategy for developing policies, legislation, programs, project, society and community’s development plans from gender equality perspective in order to benefit both women and men.Three levels of gender mainstreaming: (i) At legislation, policies, strategies level; (ii) At organization, department level; (iii) At household, family level.At legislation, policies, strategies level:

- Conducting survey, assessment on different needs, characteristics of men and women. - Build laws and policies bases on the specific characteristics of each gender to create

equality opportunities for men and women and enhance gender equality.- Build policy priorities for women or men to ensure equal opportunities for each genderExample: Labor law: Time to work, regulation for maternity and retirement

At organization, department level:- To ensure gender equality in organization and department and- Give the equal chances for men and women.- Enhance knowledge, awareness and sensibility of staffs on gender

At household, family level- Build family equality bases on mutual respect, shared work, the rights and equality in

the enjoyment of their spiritual and material value. - Wife and husband are equal in making decisions

Page 36

2.2 Why do we need gender mainstreaming?

What gender analysis will provide:• Analysis of the division of labor and access and control of resources• Understanding of gender relations and their Implications for development policy and

implementation • Specific gender disaggregated statistics• A Review of Women’s Priorities, Women’s Practical Needs and Strategic Interest and

ways to address them• A Review of Social, Economic, Political Power Dynamics

Absence of gender analysis proposes high risk of program failure, less success or reinforces inequity2.3 Objectives of gender mainstreaming in projects- Reduce gender inequality - Ensure women and men’s specific needs are satisfied, that they benefit from the project and - Ensure that the project impacts positively on both men and women’s life;- Create the conditions for the equitable access of men and women to project resources and

benefits; - Create the conditions for the equitable participation in project implementation and

decision making processesSome examples:• A gender analysis of health program will inform you how inequalities disadvantage

women’s health, the constraints women face, ways to overcome constraints. • A gender analysis of women worker’s situations, their needs, work places, wages, market

trends will provide practical information to advocate for all (women and men) worker’s rights

• A gender analysis of product supply chain will tell you women’s involvement at different stages in the supply chain so to increase their visibility and gain economic benefits

• A gender analysis of water project will inform you where women collect water, what should be done to increase women’s access to safe water

2.4 Tools and strategies for gender mainstreaming2.4.1 Basic strategies:a. Women’s empowerment

Women participate with men at five levels: (i) Discuss; (ii) Know; (iii) Implement; (iv) Monitor; (v) Make decisionTheir right to access the opportunities and resources:• Micro-credit projects• Funding for women in household economics

b. Capacity building- Approach to development goals, usually refers to skills improvement and capability

enhancing for individuals as well as communities in developing societies so that they can tackle their issues and problems:

- Participating in technical production training

Page 37

- Open vocational training for rural women

c. Equal opportunities- A fair division of study, recruitment, training, promotion and power using of individuals

regardless their gender- Example: recruitment policy, training, promotion opportunities regardless gender

2.4.2 Temporal Corrective MeasuresTemporary measures to promote gender equality to shorten the gender gaps and will be finished once the goal of gender equality achieved.a. Gender mainstreaming

- Gender mainstreaming is a strategy for gender equality accepted globally.- It is not a goal, but a strategy, approach, a way to achieve the gender equality goal

b. Gender budgetingGender sensitive analytical activities on Government budget or public expenditure

- Not focus only in budgets which are directly related to gender issues or women’s issues- Cover all budget aspects to indentify whether the budget allocation will impact differently to

women and men? If any, how will it impact?2.5 Gender analysis

a. Tools for gender analysisGender analytical frameworks analyze

- Labor division (role and responsibility of each gender)- Access and control of power at all levels of men and women- Basic and Strategic Gender interests/ needs of men and women

Gender indicators- Gender indicators measure gender-related changes over time. Gender indicators can refer

to quantitative indicators based on sex disaggregated statistical data or qualitative changes– for example, increases in women’s levels of empowerment or in attitude changes about gender equality.

Example: income of men and women, time to work of men and women- Measurements of gender equality might address changes in the relations between men

and women, the outcomes of a particular policy, programme or activity for women and men, or changes in the status or situation of men and women, for example levels of poverty or participation

b. When need to conduct gender analysisGender analysis need/can be conducted in some/all periods of a project/programe, includes:• Problem Identification/Need assessment• Project planning • Implementation; and• Monitoring and evaluation

c. How to do gender analysis- Collect Relevant Data: Sex–disaggregated information for analysis (Who does what?

Gender roles, responsibilities, priorities of men and women both within and outside the household? Who has what? Who controls what?)

Page 38

- Identify Relevant Gender Issues (women’s and men’s practical needs and strategic

interests).- Understand the institutional, economic, social, and political contexts (What are the

differences, constraints, influences, power dynamics between women and men?)- Understand the priorities and needs of both men and women affected by the project (what

do they need/want?).

2.6 What to expect in a successful gender mainstreaming project

- Equal participation by men and women in the decision-making processes for setting priorities and allocating resources;

- Equal access to and control over society’s opportunities, resources and development outcomes by women and men;

- Equal recognition and status for men and women;- Women and men enjoy equal human rights;- Equal improvements in and standards of quality of life for women and men

2.7 Basic principles in gender mainstreaming

It is necessary to specify the gender differences

- There is appropriate and reliable method in monitoring the project process.- Assumptions that issues or problems are neutral from a gender-equality perspective

should never be made;- Gender analysis should always be carried out.  - Clear political will and allocation of adequate resources for mainstreaming- Gender mainstreaming requires that efforts be made to broaden women's equitable

participation at all levels of decision-making - Mainstreaming does not replace the need for targeted, women-specific policies and

programmes, and positive legislation

Page 39

Lecture 6-7: Measuring and reporting first and second level of results 1. Overview

RIMS (Results and Impacts measuring system) is a system for measuring and reporting the results and impacts achieved by the projects.The system looks at three levels of results:

(i) First level (Outputs)(ii) Second level ( Outcomes)(iii) Third level (Impacts)

a. First level results- First-level results correspond to the project activities and outputs - measure financial and

physical progress; - First-level results are mostly quantitative;- First-level results are reported on an annual basis.àThese indicators are measures of results at either the activity, or output levels of the logframe.

à Project outputs can be measured through simple quantitative indicators.

b. Second level results

- Second-level results correspond to project outcomes - measure improved functionality or behavioral change.

- Second-level results depend on the characteristics and objectives of the project. Measuring outcomes means analyzing changes in the behaviors of households and individuals, changes in the performance of groups and institutions,

à Various methods can be employed for measuring these changes: studies, participatory approaches, questionnaire-based surveys, focus group discussions, etc…c. Third -level results- Third-level results correspond to project impact - measure the combined effects of the

first and second level results- Third-level results are quantitative- Third-level results are measured at three points during project life (benchmark, mid-term

and completion).- These refer to the goal or objective level of the project logframe.2. Why to measure and report the first and second level results?- The findings of the analysis should be used to: (i) Define corrective actions; (ii) Make

decision on improving the project strategy.- The information can provide a useful basis for analysis of project performance and

contribute to decision-making.- Comparing targets of RIMS first-level results with actual achievement may lead to an

analysis of factors that positively or negatively affected project implementation.- The analysis of RIMS second-level results provides more in-depth information on the

changes occurred at the level of beneficiaries, households, communities or institutions.

Page 40

3. Collecting information and reporting first and second level results

In order to assess whether first-level results have been achieved, a suitable system for gathering and managing information on project outputs needs to be included in the M&E activities.

This may require the identification of information exchange systems between the project management and service providers or other stakeholders.The RIMS first and second level results have been grouped in the following categories:

• Natural resources (Land and water)• Agricultural technologies and production• Rural financial services• Markets• Rural enterprise development and employment• Policy and community programming• Social infrastructure• Total outreach

Gender perspective is integrated in these categories a. Selection of first level results• The selection of first-level indicators is made on the basis of specific project characteristics.• Only the indicators related to the project should be reported• The selection of first-level indicators should be grouped according to project

componentsTable 16: Example for 1st level results

Categories 1st level results

Natural resources (Land and water)

- Number of people trained on managing infrastructures (divided by gender) - Groups managing infrastructures which have female leader

Agricultural technologies and production

- Number of extension staffs trained on (divided by gender)- Number of people trained on agricultural technologies (divided by gender)

Rural financial services

- Number of people participating in the group savings and credit which are established or strengthened by project

- The group savings and credit by women leaders

Rural enterprise development and employment

- Number of people trained (divided by gender)- Number of enterprises to access financial services

b. Selection of second level results• Identified on the basis of the project characteristics• Only the indicators related to the project should be reported• Some RIMS second-level results can be traced back to first-level results

Table 17: Example for 2nd level results

Page 41

Categories 2nd level results

Natural resources (Land and water)

- Sustainable development ability of the groups managing infrastructure which are established or strengthened

- The effectiveness of natural resource management (by gender of the household head)

Agricultural technologies and production

- Efficiency: Improving the performance of service providers (extension service)

- Sustainable development ability of the groups which established or strengthened (by gender of the household head or participants)

Rural financial services

- Sustainable development ability of the savings and credit groups which are established or strengthened

- Efficiency: improving the accessibility to financial services of the poor (divided by gender of the household head)

Rural enterprise development and employment

- Efficiency: creating employment opportunities (divided by gender)- Sustainable development ability of the business

4. Measuring first level resultsa. Measuring tip

Tip 1: For indicators about rehabilitation of infrastructure and facilities• Indicator: For indicators about rehabilitation of infrastructure and facilities (Irrigation

schemes, livestock water points, fishponds, roads, others…)• Measurement: Simple Counting (only fully constructed infrastructure and facilities

should be counted)• Data collection: Beneficiaries, local authorities, technical documents, information

contained in the payments and financial records of Project managementTable 18: Example for indicator about rehabilitation of infrastructure and facilities

Indicators Unit Definition

Road constructed or

rehabilitated

Hectares The total number of kilometers of road constructed or

rehabilitated in the reviewed period of time

Tip 2: For indicators about training• Indicators: For indicators provide information on the number of people (disaggregated

by sex and ethnic, poor and non-poor) trained by topic of the project (infrastructure management, natural resources management, crop production practices and technologies…)

• Methods: CalculatingData collection: depends on how the training is implemented

• Planning: – Planning target values of these indicators (estimating the number of people

expected to participate in training)

Page 42

– estimated when budget decisions are taken– The comparison of expected and actual number of training participants allows

assessing the relevance of the training topics• Classification: The number of people trained should be reported on the basis of the

classification and definition provided in each of the above indicators

Table 19: Example for indicators about training

Indicator Unit Definition

People trained on infrastructure management

Males/ Females

This is the number of men/women participated in training about techniques for management and/or maintenance of rural infrastructure.

Staff of financial institutions trained

Males/ Females

The number of trained female and male officials of financial institutions, including those working for any financial institutions (commercial banks, micro-financial institutions, credit group, etc)

Tip 3: For indicators about group formation/strengthening

• Indicators: For indicators provide the number of group formation/strengthening (savings and credit groups; marketing groups; community groups, common interest groups…). For each group, the number of people in these groups (sex disaggregated) and the number of these groups with women in leadership positions should be reported.

• Planning: the number of groups expected to receive project support and the number of people (sex disaggregated) in these groups and the proportion of groups with women in leadership positions.

• Methods: Counting and calculatingTable 20: Example for indicators about group formation/strengthening

Indicator Unit Operational hints

Groups involved in NRM formed/strengthened

Number The number of natural resources management groups that have been formed or strengthened by the project

Community groups with women in leadershippositions

Number Shows how many of the community groups are headed by a women or are characterized by women in leadership

Tip 4: Access to services• Indicators: For indicator providing the number of people, groups or enterprises access

the services facilitated by the project• Planning: estimating the demand for a given service within a given period of time• Data collection: a system of information exchange should be established between

project management and the institution, agency or enterprise.•

Page 43

Indicator Unit Definition

People accessing to

extension services

Males/

Females

This is number of males and females (farmers and livestock

owner) that during the reviewed period of time have

accessed the advisory services facilitated by the project.

Training should not be counted under this indicator.

b. Reporting first level results• The first level results is reported annually at the end of the calendar year of effectiveness• The reporting of RIMS first level results includes 2 sections:

– First section refers to the annual expected results, its actual value for the period under review and how this compares with expected results.

– The second section looks at the cumulative value of each indicator and how this compares with expected results

Table 21: Example of reporting first level

Indicator Unit

PY 1 Cumulative

AWPB Target

Actual % AWPB Appraisal Target

Actual % Appraisal

Market facilities constructed

Number 30 20 66% 50 20 40%

Indicator Unit

PY 1 PY2 Cumulative

AWPB Target

Actual

% AWPB

AWPB Target

Actual%

AWPBAppraisal

TargetActu

al

% Appraisal

Market facilities constructed/ rehabilitated

Number

30 20 66% 10 10 100 50 30 60%

5. Measuring and reporting second level resultsMeasuring second level results• The achievement of second-level results is measured by a rating scale made on the

basis of the project-specific data information• Projects choose the methods for measuring second-level results. The methodology

may be flexible and consist of qualitative and quantitative methodologies.• Sources: project M&E system, survey, official statistics and reports, annual

stakeholders’ workshop…Table 22: Rating scale for second level result

Score Effectiveness Assessment Sustainability Assessment

1 Highly Unsatisfactory. Very Weak.

Page 44

2 Unsatisfactory. Weak.

3 Moderately Unsatisfactory. Modest.

4 Moderately Satisfactory Moderate

5 Satisfactory Strong

6 Highly Satisfactory Very Stronga. Measuring Tip

Tip 5: For measuring the sustainability of Infrastructure• Methods: A list of factors and guiding performance questions that should be taken into

consideration in assessing whether the infrastructure is likely to be sustainable after the end of project support.

• Collection: Data can be gathered from various sources (institutions or groups managing the infrastructure can provide important insights on the factors affecting sustainability, assessment based on studies of specialists…)

Table 23: List of performance questions for each factor

Aspect Factor Performance question and dataFinancialUsers association Ability to collect user

feesAre mechanisms in place?Do members pay regularly?Are funds for maintenance available?

National/local authorities

Ability to finance Shortfall between actual and budgeted expenditures?

Willingness to finance

Likely government priorities

InstitutionalUsers Association Sustainability Frequency of conflict; exit rate

Will the association continue to exist without external project finance?

Capacity Are the skills required for infrastructure management in place?Are members adequately trained for maintenance?Are equipment for maintenance and management available?

National/local authorities

Capacity Are the skills required for infrastructure management and/or supervision in place?

Existence Is infrastructure management/supervision under a special unit? Will it continue to exist?

Technical

Page 45

Aspect Factor Performance question and data

Design Soundness Are the structure sound? Do they have structural problems?Where high-quality materials used for construction?

Operation and maintenance

Ability Do those responsible have the required skills for operation and maintenance?

Environmental Are the environmental consequences undermining the sustainability of project benefit?Is the location at risk of erosion?

SocialUse Are people using the infrastructure?

Participation Are beneficiaries involved in maintenance and management?

Tip 6: For measuring the sustainability of groups• Methods: a set of indicators providing information on the performance and functioning

of the groups (or a sample of them) should be identified to assess whether the groups formed/strengthened by the project are likely to be operational after the end of project assistance

• Framework of factors and guiding performance questions that can be used for formulating this assessment:

Table 24: Framework of factors and guiding performance questionsAspect Factor Performance Questions and DataInternal Social support Number of members that do not renew membership

Number of new membersNumber of intra-group conflictsNumber of meeting held

Financial Financial self-sufficiencyCapacity to collect fees

External Support from partner institutions

Are partner institutions willing to support groups

b. Reporting second level results

• Second level result is reported on the basis of a rating scale. These results should be used to identify corrective actions, therefore increase the likelihood that development objectives will be achieved.

• RIMS second level results should be provided at the time of Mid-term Review and the ratings should be updated every year on the basis of new data and evidence available.

Page 46

Lecture 8: Steps to conduct a surveyIn order to conduct a survey, it is necessary to undertake 12 following steps:

(1) Define objectives;(2) Assess resources;(3) Determine methods;(4) Designing questionnaire, interview guide and pilot;(5) Selecting the sample; (6) Enumerators’ training and field testing of the questionnaire;(7) Logistical planning;(8) Data collection;(9) Data entry; (10) Data analysis; (11) Writing report;(12) Communication – Dissemination

1. Step 1: Define objectives of the surveyIn order to define objectives of the survey, the following questions should be asked:

(i) Why a survey is needed?(ii) Who are the stakeholders?(iii) What is the population? (iv) What issues are studied?

2. Step 2: Assess the resources Internal resources:- Budget- Staff: (i) Are staffs qualified?; (ii) Do staffs care about the project?- Available equipments and facilities- Time line External resources

If the internal resources are not available or qualified, it is essential to look for external resources.There are many types of external resources could be mobilized. Some services are free of charge, some are not.3. Step 3: Determine methodsMethods: depending on the approach to objects to collect information, the interviews have different forms of expression. There are three popular interview methods:

(i) Face to face interview;(ii) Interview via phone;(iii) Indirect interview

Select the most appropriate method: Cost and accessibility are the two most important criteria to determine survey method. Besides, staffs’ skills, internal resources and time also could affect the determination of survey method.The best method is the one that:

Page 47

(i) Have highest response rate with a specific objective;(ii) Most convenient to the respondents;(iii) Meet with the timeline (mail survey usually takes more time);(iv) Budgeted;(v) Appropriate with staffs capacity and other resources.

Table 25: Advantages and disadvantages of methods of administering survey questions

Methods of Administration

Advantages Disadvantages

Through the Mail (post): This method involves mailing the questionnaire to predetermined respondents with a cover letter. Generally used when there is a large number of geographically dispersed respondents.

- Wider access and better coverage

- Provides anonymity- Relatively low cost- Large sample size- Respondents complete

questionnaire at own pace

- Questionnaire must be simple- Low response rate- Points of clarification are not

possible- Follow-up of non-response is

difficult

In Person: This requires face-to-face contact with respondents. Generally, makes use of small samples to canvass opinions and when dealing with sensitive issues.

- Establish empathy and interest in the study

- Can probe complex issues- Clarify respondents

queries- High response rate 

- Expensive in time and cost- May lead to interview bias- Difficult to obtain wide access- Relatively small sample size

Over the Telephone: This is a form of personal interviewing which is used to obtain information quickly. Generally used to gain access to respondents that are geographically dispersed.

- Provides personal contact- Wide geographic coverage- Easy and quick access- Can be done with the aid

of a computer 

- Short interview time- Limited to telephone owners- Can be expensive

Electronic: This is administered via the intranet and Internet through the use of email. It is becoming the most popular method for collecting data.

- Easy to administer- Low cost- Global reach- Fast capturing of data an

analysis

- Loss of anonymity- Can be complex to design and

program- Limited to computer users

4. Step 4: Designing questionnaire, interview guide and pilotStep 4.1: Design the questionnaireSteps to design questionnaire:

1. Study project documents;

Page 48

2. Identify the indicators, goals;3. Study the template questionnaire of IFAD;4. Develop new questions;5. Structuring the questions;6. Questionnaire presentation.

Design Gender sensitive questionnaire- Not only head of household, rather women and men both of a household should be surveyed;- Ask about issues that affect womenStep 4.2: Develop an interview guide Interview guide provides detailed instructions to help enumerators understand:

(i) Purpose of each question; (ii) How to raise the question; (iii) How to record the answer; and (iv) How to deal with different situation raised during the interview.

Step 4.3: Pilot (Pre-test)Pre-test the questionnaire: The purpose of pilot is to assess the feasibility of the survey and consider the issues in the questionnaire need to be revised. The pilot is expected to answer such questions as:

(1) Does each question measure what it is supposed to measure? (2) Are all the words understood? (3) Do all respondents interpret the question in the same way? (4) Are all response choices appropriate? (5) Is the range of response choices actually used? (6) Do respondents correctly follow directions? (7) Does it create a positive impression that motivates people to respond? (8) How long does it take to complete?(9) Does it collect the information you want?

5. Step 5: Selecting the sample1

Sampling methodology The sampling method applied to IFAD projects is usually two-stages sampling method: (i) stage 1- Select villages and (ii) stage 2- Select the householdsStand-by households: The households selected in advance may be not available for interview. In this case, it is necessary to have stand-by households to replaceRespondent selection: Ensure gender balance of respondents. In total selected households:

+ 30% household will interview women (wife)+ 30% household will interview men (husband)+ 40% household will interview women or men

6. Step 6: Enumerators’ training and field testing of the questionnaireStep 6.1: Recruiting enumerators, team leaders and supervisorsData collection team should have female and male members equally.

1 The detail steps to select sample is presented in the lecture 10

Page 49

Both male and female team members should be trained on ‘how to ask gender related and other sensitive questions to respondents’ The requirements and duties of enumerators:

o Coordinate with other members during fieldwork;o Complete all parts of the questionnaire, and checking the filled-in questionnaire;o Re-interview if requested by the supervisor; o Fully take part in the evaluation sessions with the supervisors and ask them for advice

if any problems occur.The tasks of the enumerator team leader

o Guarantee the best preparation for the field test;o Delegate assignments for the enumerators;o Coordinate all activities of the team, examining the quality of collected

questionnaires;o Keep closely in touch with supervisor when carrying out the field work;o Set interview goals every day for the enumerator team;o Provide logistics for the survey;o Monitor the progress of work, reporting to supervisor

The duties of supervisors:o Supervise the interviewing process;o Understands gender and can ensure that gender is taken care of during entire data

collection process o Ensure the progress of work;o Check all answer sheets;o Review and learn by experiences everyday as well as disseminating the schedule and

plan for the following day;o Be responsible for retaining all questionnaires and handing them over to the survey

team leader for data entry.Step 6.2: TrainingThe training will cover following issues:

(1) The purposes of the Survey and steps to conduct;(2) Sampling method;(3) Structure of the questionnaire; the purpose and content of each question;(4) Interview skills, how to raise the questions and how to record the answers; (5) The role and responsibility of enumerator, team leader, supervisor;(6) How to enter and analyze data

Training materials(1) Lectures;(2) Questionnaire; (3) Interview guide;(4) Data entry form and software for entering and analyzing data;(5) Guidance for data entry and analysis

Page 50

7. Step 7: Logistical planning

One of the factors that make a survey successful is good logistics. Here are the logistical preparations for the survey:

(1) Scheduling the survey(2) Local assistant staff(3) Village Head(4) Village Guide(5) Interpreter(6) Transportation(7) Survey Materials(8) Budget estimation

8. Step 8: Data collectionData collection on a large scale is a main assignment of whole survey because this task decides the quality of survey. Besides, supervision also plays a very important role in the success of the survey.Interview female respondents- Before implementing the interview, enumerators should find out the family

circumstance through village header or guide. - Enumerators should understand cultural differences, ethnicity differences to avoid the

taboo; it can make the collection information difficult.- Enumerators should know where and when to visit women survey participants

(example: avoid the time when the women are busy for their farming…).- In the interview, enumerators should avoid the presence of the third person; it can

affect the women’s answer.9. Step 9: Data entryStep 9.1: Data entry form designThe data entry form can be designed right after the questionnaire is finalized. Much software can be used to enter data; however it is recommended that CSPro should be used for data entry because of its strong advantages. Step 9.2: Data entry: Data cleaningà Entering dataà Compare dataà Concatenate databaseà Export to SPSS/STATA/EXCEL.10. Step 10: Data analysisFor data analyzing, it is recommended using SPSS because of its simpleness. The results will be inputs for report writing.Gender analysis: Conduct a gender analysis to see the differences between women’s and men’s access to resources, control over resources, decision making, time contribution, labor contribution, other women empowerment issues etc à Disaggregate all data by sex of household head; disaggregate data by sex of respondents when necessary 11. Step 11: Report writingThe report must address following requirements: In terms of content

- The report should cover the project’s objectives, analyzing the identified indicators;- The report should clarify project activities which have been implemented;

Page 51

- The report should emphasis the different results of the project, the comparison

between the treatment group and the control group, as well as different targeted groups of the project, and gender perspective

- The report should discuss differential impacts on women and men - The emphasis should be on the meaning and implications of the data analysis for the

project, not on the methodological aspects;- Ensure the clarity and coherence of the structure, content, vocabulary and logic

between paragraphs and sections.In terms of presentation

- In addition to the main content, the report should also have a list of acronyms, a table of contents, and list of annexes;

- The structure of the sections should be clear and logic;- Surveys results should be presented first and foremost through charts/graphs;- When preparing charts/graphs, be careful with the use of colors.

12. Step 12: Communication – Dissemination

Page 52

Lecture 9: Questionnaire design1. What are questionnaire?

There are various definitions of questionnaire:• “A set of questions arranged in a certain structure to collect information from targeted

group to answer one or several of research questions”.• “A set of questions particularly used to address some statistically typical issues by

gathering information in a survey.”General structure of a questionnaire:

- Name of the questionnaire- Introduction- The basic instructions to complete the questionnaire (if required)- Questions- Acknowledgements- The management

Why use questionnaire?Questionnaire is used to:

i. Assess the need for investment and development, and the differences among project targeted areas;

ii. Measure outcomes of project and intervention programmes;iii. Assess the beneficiaries’ satisfaction;iv. Assess the project impacts in general and impact on women and men in particular.

2. The importance of questionnaire- Questionnaires are the most important tool of a social investigation.- Questionnaires reflect the research issues.- The quality of a questionnaire determines the quality of a survey.- Designing a high quality questionnaire helps: (i) Collect most accurate data; (ii)

Increase response rate; (iii) Reliable and valuable; (iv) Save time and money.Low quality questionnaires: non response may affect the questionnaire validity just as same as missing data (or obtaining inaccurate data).Low response ratesà Reduce the effectiveness of sample size à Reduce the research power à Reduce the accuracy of final results3. Steps to design questionnaireIn order to design questionnaire, six steps would be undertaken.Step 1: Study project documentsStep 2: Identify the indicators, goalsStep 3: Study the template questionnaire of IFADStep 4: Develop new questions, gender questionsStep 5: Structuring the questionsStep 6: Questionnaire presentation3.1 Study project documentsStudy project documents to understand the project’s objectives, activities, the logical framework and indicators for monitoring and evaluation.

Page 53

3.2 Identify the indicators, goalsIdentify the indicators which can be collected through the survey. Make a list of indicators that the survey needs to assess.3.3 Study the template questionnaire of IFADStudy the template questionnaire of IFAD to identify the sections and questions that can be adapted directly to the project; determine which questions need to be adjusted and added.3.4 Develop new questions, gender questionsDevelop new questions to collect information for the indicators. Gender issues should be taken into account when designing the questions.+ Project impacts on gender issue;+ Gender difference about participating in project activities.Types of survey questions

o By the content: (i) Questions about activities, facts/events; (ii) Questions about psychological status or attitudes; (iii) Questions about knowledge/perception

o By the function: (i) Psychological questions; (ii) Filter questions; (iii) Reflective questions; (iv) Information questions

o By the expression of the answer: (i) Closed questions; (ii) Open questions; (iii) Half-open questions

o By the expression of the question: (i) Direct questions; (ii) Indirect questionsType of measuring scales

o Nominal scale: E.g. Male/female; Provinces: Bac Kan/Ha Tinh/Tra Vinh/Tuyen Quango Ordinal scale: E.g. Very poor – poor – average – rich – very rich; like – normal – dislikeo Interval scale: E.g. range of income, range of age.o Ratio scale: E.g. Percentage of male/female, ethnic groups

When designing additional questions, keep in mind the following points:o Each question should have a specific purpose; avoid questions that overlap or

duplicate each other or questions with an unclear purpose;o The question must not give any hints, bias or lead respondents to any answers;o Have clear instruction for answering the questions or skipping questions;o Wording with local language, avoid using acronyms or jargon that are multi-meaning,

unclear or abbreviated. 3.5 Structuring the questionsStructuring the questions into sections is in a logic order. The questions are divided into sections, each of which focuses on a single theme on project’s activity or objective.3.6 Questionnaire presentationWhen present questionnaire, it should be noted:

o Title; introduction (enclosed with intro letter); o Contact;o Minimal types of question;o Instructions;o Avoid horizontal & vertical structure;o Avoid overuse of boxes, tables, and lines;

Page 54

o Check the format.

Introduction should include:o Who is conducting the survey;o The topics covered in the survey;o Statement of confidentiality;o Statement that the time length of the survey depends on the research mode, topics,

population.Check the format of questionnaire:

o Number all the questions sequentially;o Use large, clear font type;o List groups of answers in vertical sequence;o Avoid double/triple-choice answers;o Be consistent with guidelines and orders of answer groups;o Don’t split question across pages;o If necessary, move the question and answers to next page;o Put specific instructions of questionnaire as needed, then the questions;o Distinguish instructions from questions;o Pre-code the questionnaire using check boxes.

4. Principles for questionnaire designo Ask what you want to attaino Understand the mind of respondents and the context of the interviewo Start with simpler questions (not sensitive)o Use proper logic. If the questionnaire is long, start with the key questionso Put questions on anthropology at the endo Put questions for past experiences prior to those for current statuso Put general questions first, then specifico Use the rule of filtering;o Use equivalent terms (avoid technical ones);o Avoid unclear terms/short terms;o Avoid multi-meaning words/terms;o Keep sentences short (if possible);o Avoid suggesting the answers (by proposing a hypothesis);o Avoid leading questions;o Avoid bias.

Requirement for a good questionnaireA good questionnaire matches following requirements:

o The questionnaire can collect sufficient information for assessing outcomes (and impact) of project;

o Collected information must present project’s objectives and indicators; o Questionnaire has appropriate length;

Page 55

o The questions must be easy to ask and answer;o Questionnaire has appropriate structure and clear format.

5. Questionnaire piloting- Preferable to test the questionnaire with those similar to targeted subjects of the

research population- Conduct pilot using the same mode for the major study- Consider cognitive pre-testing

The purpose of pilot:(v) Develop and test the research tools(vi) Assessing the feasibility of a survey against sample size(vii) Identify logistical problems of the questionnaire

- Prepare first draft and pre-test- Distribute questionnaire to small sample- Evaluate responses & re-draft

Checklist items:1. Does each question measure what it is supposed to measure? 2. Are all the words understood? 3. Do all respondents interpret the question in the same way? 4. Are all response choices appropriate? 5. Is the range of response choices actually used? 6. Do respondents correctly follow directions? 7. Does it create a positive impression that motivates people to respond? 8. How long does it take to complete?9. Does it collect the information you want?

6. Summary of steps to design the Questionnaire1. Decide what information is needed2. Search for existing questions3. Draft new questions/revise existing ones4. Prepare interviewer instructions for pilot test5. Pilot test6. Revise questions based on respondent & interviewer comments7. Pilot again, if necessary8. Prepare interview instructions9. Training10. Field test11. Finalizing questionnaire and guidance after training and field test

Page 56

Lecture 10: Sampling method1. Definitions1.1 Population The population includes units (elements and particular phenomenon) that need to be observed and analyzed (quantitative analysis) to find out their rules and nature in specific time and space. E.g. The study of the impacts of a project in 2012 in Tuyen Quang provinceà the population consists of all households in the project area of Tuyen Quang in 2012. 1.2 Sample A sample is a subset of a population that is randomly selected to be representative of the population. Sample selection is the selection of a specific number of units in the population that are representative of the population by using the different sampling methods. E.g.: In order to evaluate the impacts of a project in a locality à select randomly 900 households among all beneficiaries of the project to participate in the survey. 1.3 Sampling frameSampling frame is the list of communities/villages (clusters) in the project/ non-project area that is used to select the representative communities/villages (clusters) in the project/ non-project area. 1.4 Sample size Sample size refers to the number of households to be included in the survey. An appropriate sample size is determined by three factors: (i) the sale of population, (ii) implementation cost and (iii) the acceptable margin of error. 2. Overview sampling method2.1 Random sampling methodThis method ensures that a target population has an equal probability of selection. This method includes the following:

- Simple random sampling - Stratified sampling- Cluster sampling

a. Simple random sampling- Used when the overall population has no distinct characteristics and is homogeneous and

it is not too widely dispersed. - Advantages: simple and time-saving. - Disadvantages: there might be difficulties in collect information because households are

scattered in a large area E.g.: Study of the impacts of a project to the households in a locality which is not too widely dispersed

- List all households and assign each household a number Method 1: Use function “Sampling” in Excel to choose a number of households in the village to participate in the survey Method 2: Using Households list

Page 57

- Assign each household in the community a number (attach head of household’s name

if necessary) and write it on a pieces of paper, the number of pieces is equal to the total number of households in the village.

- Put them in a jar or bag, mix them and draw out a number of pieces of paper.- The number drawn represents the household numbers from the list, complete the list

of households. b. Stratified sampling

- This method is used when the overall population of project beneficiaries includes many group or strata with very distinct, mutually exclusive socio-economic characteristics

- Advantages: simple, time-saving and implementation cost-saving.- Disadvantages: potential systematic errors

E.g.: Study of smoking status of men in Hanoi - Population: all men in Hanoi - Stratified sampling based on the age of men - Divide population into 3 groups of age: under 18 years old, from 18 to 30 years old and more than 30 years old. - With each group of age, select a certain number of men to take part in the survey.

c. Cluster sampling- This method is used when there are difficulties in geographical conditions, the

population can be divided into many areas.- Advantages: it is not necessary to list all elements, reduce cost on finance and human

resources.- Disadvantages: the sample cannot be the best representative

E.g.: study of the impact of the project - Population is divided into 20 groups- Make the list of 20 groups - Select 5 groups by using random sampling method - All households in 5 selected groups are selected in order to participating in the study

2.2 Non-random sampling methodOnly some members of the beneficiary population have a chance of selection, while other will be excluded a priori from participation in the survey.

- Convenience sampling - Purposive sampling- Respondent-driven sampling

a. Convenience sampling- The sample is selected conveniently or accidentally that is easy to reach or close to hand.- Advantages: save time and reduce cost.- Disadvantages: the representation of sample is not ensured, leading to low preciseness

E.g.: Study of the impact of building infrastructure on accessing market of households in the project area

à Population: all households in the project area à Researcher selects the households which are near road and easy to access

b. Purposive sampling

Page 58

- This method involves choosing the sample based on the best judgment on who would be

appropriate for the study.- Advantages: choose the appropriate sample for the study.- Disadvantages: the representation of sample is not ensured

E.g.: Study of the utility of the training provided by the project à Researcher selects the households succeed or fail in applying knowledge gained in the training course to participating in the studyc. Respondent-driven sampling- Advantage: It is very useful and especially convenient for the study of hard-to-reach groups.- Disadvantages: the sample may not be representative, so conclusion cannot be validly

made about the population3. Sampling method in project funded by IFADIFAD impact surveys usually apply two-stage sampling method: The first stage is to select villages:

In the RIMS+ baseline and end-line survey, there are 30 villages in the project area and 30 villages in the non-project area selected In the Annual Outcome survey, there are 20 villages in the project area and 20 villages in the non-project area selected The second stage is to select households:

In the RIMS+ baseline and end-line survey, there are 30 households selected in each selected village.In the Annual Outcome survey, there are 20 households selected in each village.

3.1 Stage 1: Select villagesa. Select villages in the project areaStep 1: Construct sampling frame

- List all villages of project communes;- Calculate the cumulative population

Example: In “TNSP Project in Tuyen Quang”, there are 25 project communes; thus, the sampling frame is the list of all villages of 25 project communes. The sampling frame is constructed as follows:

The sampling frame of TNSP in Tuyen Quang

No District Commune Village Population Cumulative population

1

Na Hang Nang Kha

Na Chao 121 1212 Bản Nhùng 54 1753 Bản Tùn 68 2434 Bản Nuầy 80 3235 Phiêng Quân 69 3926 Phiêng Rào 58 4507 Không Mây 72 5228 Nà Noong 111 6339 Nà Chang 74 70710 Nà Reo 49 756

Page 59

No District Commune Village Population Cumulative

population11 Nà Kham 119 87512 Nà Khá 109 98413 Nà Vai 82 1,06614 Lũng Giang 52 1119… … … … … …233

Son Duong

Dại Phu

Lũng Hoa 110 18,651234 Hoa Lũng 81 18,732235 Thái Sơn Đông 70 18,802236 Thái Sơn Tây 62 18,864237

Hop Hoa

Ninh Hoà 190 19,054238 Thanh Bình 206 19,260239 Thanh Sơn 98 19,358240 Đồng Chùa 103 19,461241 Núi Độc 100 19,561242 Đồng Phai 97 19,658243 Cầu Đá 222 19,880244 Đồng Giang 153 20,033245 Đồng Báo 84 20,117246 Việt Hoà 43 20,160247 Tân Trào 206 20,366248 Tân Dân 116 20,482

TOTAL 20,482Step 2: Calculate sampling interval (SI)

SI= total population/number of selected clustersE.g.: SI= 20482/30 = 683

Step 3: Choose the random number Select the random number by using the Excel function rand/randbetween Type: = Rand()*SIE.g.: Rand()*683= 236

Step 4: Select village The first cluster corresponds to the village that has a cumulative number of households equal to or greater than the random number.

- The second cluster corresponds to the village that has a cumulative number of households equal to or greater than the random number, plus the SI

- The 3rd cluster has the larger cumulative number of households closet = the cumulative number of households of the second cluster + SI.

- Similarly, we will select the last cluster corresponds and complete the list of households.

b. Select non-project villages

The use of control group- The control group will represent what the treatment group would have been like if

there was no project/intervention.

Page 60

- And instead of comparing before- after of the treatment group only, compare the

difference in before-after between the treatment and control groups.Figure 3: The use of control group

Sample size of the control group- In the baseline and end-line survey, the sample size of the control group is 300

households (10 villages and 30 households in each village) - In the annual outcome survey, the sample size of the control group is 200 households

(10 villages and 20 households in each village) Select village in the non-project area

Method of selecting village in the non-project area Step 1: Select 10 communes out of the project area, that they have a socio-economic

and geographically condition with target communes Step 2: Select 10 villages in 10 selected communesIn order to select 10 villages in the non-project area, repeat the steps in selecting villages in the project area.

- List all villages of 10 communes- Calculate SI- Select a random number- Select villages

3.2 Stage 2: Select householdsThere are 3 methods used to select households in project and non-project area Method 1: Systematic sampling

Step 1: List all households of each village in excelStep 2: Calculate sampling interval: SI = total number of households/selected householdsStep 3: Select a random number (R)Use function ‘Rand’ in Excel. Type (= rand()*SI) in a cellStep 4: Select household- Rth household in the list is the first selected household.

Page 61

- Add the sampling interval –SI to select 2rd household- Similarly, we will specify the order number of the last household.

Method 2: Simple random samplingStep 1: Assign each household with a number in ExcelStep 2: Using function “Sampling” in Excel to choose a number of households for the survey.

Method 3: Simple random sampling without using Excel softwareOption 1: Using Households list - Assign each household in the community a number and write it on a piece of paper,

the number of pieces is equal to the total number of households in the village.- Put them in a jar or bag, mix them and draw out 30 pieces of paper. - The number drawn represent the household numbers from the list, complete the list of

30 households Option 2: Random Walk- Identify the approximate centre of the village. A pencil or bottle is spun to select a

random walking direction (also called a transect line). - Count the number of households encountered along the transect line between the

centre and the end of the village. This number is divided by total number of households needed to determine the sampling interval (SI).

- Select a random number between 1 and total number of households. The random number corresponds to the first interviewed household

- Continue the process until the required number of households has been interviewed Stand-by householdsThe households selected in advance may be not available for interview. In this case, it is necessary to have stand-by households to replace

- In the Annual Outcome Survey, the number of stand-by households should be equal or half of needed households- In the baseline or end-line survey, the stand-by household can be 10 or 15 households.- The stand-by households are selected by using the method applied in official household selection

Gender - select the respondentsIn total selected households:

+ 30% household will interview women (wife)+ 30% household will interview men (husband)+ 40% household will interview women or men

Selecting method: Method 1: Select randomly from the list of households (before going to the field) à proposed the village head to inform the selected object to participate in the interview. Method 2: Provide the list of the selected households (before going to the field) à propose the village head to identify households with men or women at home and select the suitable intervieweeMethod 3: Select and identify in the field.

Page 62

Lecture 11: Introducing the Annual Outcome Survey1. What is the annual outcome survey?

The Annual Outcome Survey (AOS) is a simple household survey that will be undertaken annually by project staffs. The purpose of the Annual Outcome Survey:

i. Measure more regularly the positive or negative changes/outcomes taking place at the household level;

ii. Provide early evidence of project success or failure;iii. Provide timely performance information so that corrective actions may be taken if

required;iv. Assess targeting efficiency

2. How and when to conduct the Annual Outcome survey? How to conduct the surveyThe survey should be implemented in conjunction with qualitative assessments. Key informant interview (KII) and focus group discussion (FGD) will be conduct to provide household-level information about “why” and “how” some outcomes were or were not achieved.When to conduct the surveyAOS should be conducted every year starting from the 2nd year of implementation.3. Steps to conduct the AOS3.1 Step 1: Designing questionnaire, interview guide and pilot a. Designing questionnaireStep 1: Study carefully project documents Step 2: Identify the indicators which can be collected through the AOS. Which gender indicators should be measured and measurable through AOS.Step 3: Identify the sections and questions in the template questionnaire of IFAD can be adapted directly to the project; determine which questions need to be adjusted and added. Which gender sensitive question is available, which is needed to be developed.Step 4: Develop new questions to collect information for the indicators. Gender perspective should be taken into account when designing the questions.Step 5: The questionnaire are divided into sections, each of which focuses on a single theme on project’s activity or objective. Determining to have a separate section for gender issues or included in other sections. In addition to the gender sensitive questions integrated in parts, there is a separate section on gender issues. This part will mention aware of domestic violence and power in making decision in the family and communityStep 6: Questionnaire format: Each section should be designed to fit on one page; Text size and style should be easy to read; Instructions for skipping questions should be highlighted and placed in a visible placeb. Develop an interview guideInterview guide provides detailed instructions to help enumerators understand:

i. Purpose of each question;ii. How to raise the question;

iii. How to record the answer;

Page 63

iv. How to deal with different situation raised during the interview

When developing the interview guide, it is necessary to take into account the following issues:- Content: indicate general rules for interviewing and recording the answers and

explain the content, purpose of each question; provides definitions, solutions to special cases and gives example if necessary.

- Presentation: Interview guide must be clearly displayed and have clear annotation (bold, italic, underline).

c. Pilot- The purpose of pilot is to assess the feasibility of the survey and consider the issues in the questionnaire need to be revised- Sample size for pilot: 5-10 households.3.2 Step 2: Samplinga. Sample size:

- Sample size of the treatment group is 200 households and sample size for the control group is 200 households. It includes 40 clusters; 10 households at each selected cluster.- The sample can be replicated every year for the AOS- Besides comparison the control group and the treatment group, we compare the women respondent and the men respondent especially for perception and opinion-related questions. Therefore, in each cluster, we will determine in advance the households which interview women and the households which interview men. b. Sampling methodologyThe sampling method applied to IFAD projects is two-stage sampling method.Stage 1: Selects clusters: Select 20 villages in the project area and 20 villages in the non-project areaStage 2: Selects households: Select 200 households for the treatment group and 200 households for the control group. Select 10 households from each selected village (incl. poor, pro-poor, and women head of households)3.3 Step 3: Enumerators’ training and field testing of the questionnairea. Recruiting enumerators, team leaders and supervisorsEach group has 01 supervisor, 01 team leader and 05-06 enumerators. It should have female and male enumerators in each group to ensure gender balance.b. TrainingThe training will cover following issues:

(1) The purposes of the AOS and steps to conduct;(2) Structure of the questionnaire; the purpose and content of each question;(3) Interview skills, how to raise the questions and record the answers; (sometimes cannot

get information when conduct an interview both men and women, have to separate them to two groups for women easy to speak out)

(4) Sampling method;(5) The role and responsibility of enumerator, team leader, supervisor;(6) How to enter and analyze data

c. Field-test

Page 64

The field-test will be conducted after all enumerators participate the training. It will be conducted similarly to the survey on large scale. It should have good preparation. After the field-test, all participants will share experiences3.4 Step 4: Logistical planningThe survey should have a good logistical planning including:

(1) Scheduling the survey(2) Local assistant staff(3) Village Head(4) Village Guide (5) Interpreter(6) Transportation(7) Survey Materials (8) Budget estimation

3.5 Step 5: Data collectionData collectionEach district will schedule the survey based on the time, survey areas, and number of enumerators. The plan should be consistent with the overall survey plan of the whole project. All enumerators will be divided into many small groups and visit households in the list to conduct the interview.SupervisionSupervisors and team leaders are responsible for the quality and the progress of the survey by supervising the interview and check the completed questionnaires. 3.6 Step 6: Data entry- Design data entry form: CSPro should be used for data entry- Data entry: Data cleaning à Entering data

3.7 Step 7: Data analysisEntered data will be exported into an analyzing software. It is recommended using SPSS because of its simpleness. Analyze and compare between:

- Male head of household and female head of household (all questions); female respondents and male respondents (selected questions only)

- The treatment group and the control group, and- Compare over the years

3.8 Step 8: Report writing The AOS report is mainly based on analyzed data in the survey. However, the report also use the information obtained from KII and FGD to have in-depth analysis and explanation for the findings of the survey. Before drafting the report, it is necessary to design a detail report outline. From the outline, a full report will be developed.The report must address following requirements:In terms of contents: The report should- Cover the project’s objectives, analyzing the identified indicators- Compare between the treatment group and the control group, between female head of

household and male head of household, as well as different targeted groups of the

Page 65

project; for some questions, it is necessary to compare or analyze data from female respondents and male respondents

- Emphasize the impacts of the projectIn terms of presentation- Ensure the clarity and coherence of the structure, content, vocabulary and logic- Have a list of acronyms, a table of contents, and list of annexes- Surveys results should be presented first and foremost through charts/graphs- The accompanying text should only highlight the most important findings from the charts

After a draft report is prepared, the PCU’s director board will organize a workshop to share the findings and collect comments on the draft report from project staffs and relevant stakeholders. After that the report will be finalized and ready for sharing at a wider community.

Page 66

Lecture 12: RIMS and RIMS+ baseline and end-line survey1. Introduction to RIMS and RIMS+ survey

RIMS (Result and Impacts Measuring System) is a comprehensive system for measuring, aggregating, and reporting on the results and impact of IFAD supported country programmes and projects;RIMS is made up of a standardized list of common indicators that could be applied across all IFAD projects and programmes:There are 3 levels of results

i. 1st level indicators (outputs): refer to project activities and outputsii. 2nd level indicators (outcomes): relate to project outcomes and reflect changes in

beneficiary’s behavior, improved performance and sustainability of groups, institutions and infrastructure.

iii. 3rd level indicators (impact): are associated with project impact on child malnutrition and household living standards

The baseline and end-line household survey are used to collect information/evidences for third-level results1.1 Background on RIMSa. RIMS survey- Sample size: 900 households in the project area- Should be done before and at the end of projectb.RIMS questionnaire

- Objective is to measure assets and child nutrition- Questionnaire is divided into three sections:

i. Section 1: Household demographicsii. Section 2: Housing, assets, and food security

iii. Section 3: Anthropometry - Little flexibility in questionnaire design and analysisc. Limitations of RIMS survey/questionnaireThere may be external events that may adversely impact acute and chronic malnutrition. Most IFAD-funded projects are multi-sectorial interventions à the scope of the standard RIMS impact surveys may be too limited to reflect the variety of impacts that IFAD-funded projects may have at the household or community level.à RIMS+ impact survey is necessary to reflect fully the impacts of IFAD projects.1.2 Changes in RIMS+

Table 26: Overview of changes in RIMS+

Changes Advantages1. Expanded questionnaire

Collect additional information, assess fully the impacts of IFAD project

2. Use of control group Better measurement of impact of project by controlling for broader changes in rural conditions

3. Additional training and Improve quality of data

Page 67

Changes Advantages

supervision 4. GPS to geo-reference households

Facilitate return to same households (panel) and better supervision of enumerators

5. Flexible questionnaire & analysis

Address information needs of the IFAD project and IFAD planning in general

a. Expanded questionnaire

Table 27: Information in RIMS and RIMS+ questionnaire

Contents RIMS New information in RIMS+ Cover page (no information) - Information on the province, commune,

village, household, enumerators, supervisors +Name and code of commune, village

A. Member characteristics

Household members: age, sex, literacy

- Ethnicity, school attendance (gender disaggregated data), & reasons for not attending

B. Housing Floor material, number of bedrooms; sources of drinking water; toilet types

- Roof material- House ownership certificate and licensee (gender disaggregated data)

C. Assets AssetsFuel for cookingTools for tilling the land

- Agricultural equipment

D. Land (no information) - Farm size- Land use certificate (gender disaggregated data)

E. Food security Period of hungry season

- Food shortage resolution (gender disaggregated data) - Daily nutrition (gender disaggregated data)

F. Crop production (no information) - Production, sales, prices of crops; input cost- Division of labor (gender disaggregated data)

G. Livestock & fisheries

(no information) - Herd size, sales, & input costs- Division of labor (gender disaggregated data)

H. Extension & market access

(no information) - Extension training: participation, attendance times, applicability (gender disaggregated data) - The highest income product, selling location and time of travel to there; seller and buyer

I. Non-farm activities

(no information) - Non-farm income- Primary source and secondary source of income - Financial management and main income earner (gender disaggregated data)

J. Credit assess (no information) - Access to credit, information on loans received

Page 68

Contents RIMS New information in RIMS+

- Borrower and the person responsible for paying (gender disaggregated data)

K. Socio-Economic Development Plan (SEDP) and infrastructure building plan

(no information) - Participating in SEDP and infrastructure plan (gender disaggregated data) - Disseminating community information- Satisfaction with local infrastructure

L. Risk & vulnerability

(no information) - Perceived risk of natural disasters

M. Gender equality (no information) - Changing awareness of family violence - Role of women in decision making process in the family and community

N. Anthropometry Age; Height and weight of child (boys and girls)

(no new information)

Gender issue is integrated in the RIMS+ questionnaire

Not only head of household (men), rather women and men both of a household should be surveyed à Compare women and men respondent especially for perception and opinion-related questions and compare women headed household and men headed household about the following aspects:

+ The length of food shortage+ Resolution for the lack of food problem + The status of malnutrition of girl and boy child+ Accessing land and the right of using land + Productivity of farming and livestock+ Income from farming and livestock+ In contribution in farming and livestock (who has to work more?)+ Who in the household usually borrow money+ Who in the household is usually responsible for repaying the loan+ Method of accessing market information and information received+ Decision power to sell products+ Applying the job trained to household economic+ The participation of female and male in activities of community, this demonstrate the confidence and activity of female+ Capacity to withstand or recover from disasters

b. Use the control group

Page 69

Table 28: Example for measuring impact of project

by using information of the control group

- The control group includes 300 households which have the same characteristics with the treatment group, only in the non-project area.

- The control group will represent what the treatment group would have been like if there was no project/intervention.

- Instead of comparing before- after of the treatment group only, compare the difference in before-after between the treatment and control groups

c. Additional training and supervisionBecause RIMS+ questionnaire is longer and somewhat more complicated, need for additional training & supervision of enumerators.Enumerators and supervisors are provided a training taking at least 6 days, including 2-day data training and 4-day survey training.Consultant will provide additional supervision during data collection, particularly important in first week of data collection.d. Use of GPS units

Main purpose is to make it easier to find household to interview in later round of surveye. Flexible questionnaire & analysis of results

The original RIMS has limitations in customizing the results for project. The RIMS+ questionnaire allows us overcome the limitation of RIMS questionnaire. The RIMS+ questionnaire can be adapted to fit with each project in each local, and the data entry form can be adjusted easily to enter data.2. How to conduct RIMS and RIMS+ survey

RIMS and/or RIMS+ survey is implemented at the beginning (as baseline survey) and the end of the project (end-line survey).There are two options for conducting RIMS+ survey:

• Option 1: For the projects already conducted RIMS baseline survey: conduct RIMS+ end-line survey when the project completes (using RIMS questionnaire for baseline survey and RIMS+ questionnaire for end-line survey);

Page 70

Beneficiary households

Non-project households

Measuring impact by comparing pre-and after of the treatment group only

Measuring impact by using information of the control group

E.g. 1 Income rises 8%

Income rises 4% due to economic growth

Suggests that project caused 8% increase in income

Actually, only a 4% increase due to project

E.g. 2 Income does not change

Income falls 4% due to drought

Suggests that project had no effect

Actually, 4% increase in income due to project

• Option 2: The projects have not yet conduct baseline survey: conduct RIMS+ baseline

survey and RIMS+ end-line survey (using same questionnaire for the two surveys). The steps to conduct RIMS survey and RIMS+ survey are similar. To conduct RIMS and/or RIMS+ survey, the following steps are undertaken:

• Step 1: Design questionnaire and interview guide;• Step 2: Sample selection;• Step 3: Enumerators’ training and field-test;• Step 4: Logistical planning; • Step 5: Data collection;• Step 6: Data entry;• Step 7: Data analysis;• Step 8: Report writing

Some notes when conducting RIMS and RIMS+ surveyIn terms of sample size: Sample size for RIMS+ survey is 900 households in the project area (30 clusters*30 households in each cluster) and 300 households in the non-project area (10 clusters*30 households in each cluster).Sample size for RIMS survey is 900 beneficiary households.In terms of logistical plan:It should have a good preparation on all activities in the survey such as: organization in training (including the field-test) and the survey; documents (questionnaire, interview guide, lectures); and equipment (weight and height scales; GPS machine).In terms of data entering and analyzingIf the RIMS standard questionnaire is used, the RIMS software will be used to enter data. If the RIMS+ questionnaire is used, given that the standard RIMS software cannot accommodate additional questions of RIMS+ questionnaire àwe will have to create a separate data-base in order to enter and analyze answers to these additional questions. CSPRO software is recommended to enter data and SPSS software is used to analyze data.Disaggregate all data by gender of head of household and gender of respondent.

Page 71

Lecture 13: Key informant interview1. Definition of key informant interviews

What is a key informant interview?A key informant interview is a dialogue between researchers and people who can provide detailed information (hereinafter called ‘interviewee’) in order to understand interviewees’ lives, experiences and perception on particular issue. This is personal interview or face to face method.When needs to conduct key informant interviews?• Key informant interviews are used in the following cases;• Research topics are new and undefined;• Being opinion polls when definitions, problems, phenomena, or subjects’ characteristics

researched are uncertain;• Need more in-depth information to maximize understanding about research topics; • Profound insights into meanings of phenomena, actions rather than their frequency;• To answer questions “WHY” and “HOW” adding more information for quantitative data.

Advantages• Interviewees are free to give opinions and views;• Researchers are active and flexible in conducting interviews depending on specific actual

conditions (subject, situation, time and interviewer’s competency, etc.) to make interviews on different ways.

Disadvantages• Because of diversified and diffused answers so that it is very difficult to synthesize;• Because key informants are often expert on research issues causing an academic gap

between interviewers and interviewees that could make what interviewers’ take-note different from interviewees’ opinions;

• It requires a certain academic knowledge and interview skill of researchers. 2. Classification of key informant interviews

Unstructured interviewThat is a method independent on predefined questions; researchers only need remember the theme interviews and do not need to use a list of topics in the interview

Semi-structured interview

Semi-structured interview is an interview based on a list of prepared questions or topics on a certain issue. However, the order and question asking may depend on the context and characteristics of respondents.

Page 72

Table 29: Advantages and disadvantages of unstructured and semi-structured interview

  Unstructured interview Semi-structured interviewAdvantages 

- Researcher can actively change the order of topics depending on the circumstances of the interview and the interviewee's answers;- An unstructured interview like a talking which makes interviewees feel comfortable and open to answer interview topics;- It is especially useful in cases of need to interview those who provide information many times, in many different contexts; - It is useful in those cases that cannot use official interviews.

- Using the interview guide will save time;- A list of questions will help identify the issues that need to crawl but still allow the flexibility to discuss emerging issues if necessary;- Easy to systematize and analyze the information obtained;- Helps interviewers not miss the contents to be interviewed.

Disadvantages 

- No sample preparation is available so that each interview is an unique conversation so it is difficult to systematize the information and data analysis;- Interviewers must have knowledge, experience and skills;- It is very easy to miss the content of interview if the interviewer does not have the skills to dig deeply into research themes;- If the interviewee is not open to interview, the interviewer hardly lead story without prepared topic.

- It takes time to research and explore to identify research topics and design appropriate questions prior to the interview. 

3. Steps to conduct key informant interviews

3.1 Identify the topics that need in-depth information

The topic identification is the first important step to conducts the others. The number of topics depends on the research objectives, time and available resources. Each key informant could be interviewed for one or more certain topics.

Examples: Community participation in local planning from assessment on participation of male

and female; Household accessibility to market information: from angle of poor household, near

poor households and wealthy households;

Page 73

Quality of extension services for the poor and ethnic minority; Quality of vocational training for poor women.

3.2 Identify the key informants Key informant must be carefully selected. Key informant is a person who has in-depth

understanding about the research topics and can provide detailed information and opinions on a specific subject based on his/her knowledge of that issue.

To identify the informants who can provide exact and appropriate information, following method could be applied:

Identifying the relevant groups from which key informants may be drawn. Consulting from different information resources, maybe from a focal point of

the research, from available documents, materials, etc. 3.3 Develop the questions by topicsThe followings are common questions used in key informant interviews:

Description questionE.g.: Can you tell me about your family members?

Opposition questionE.g.: Compared to 05 years ago, the main crops of the family changed? Why have those changes?

Question on perspective /valuesE.g.: what can the group that the family/family members participate in contribute in the development of production?

Question on knowledge:E.g.: What do you know about local projects? How can you know/get that information?

Question on feelingE.g.: How do you think about the project to contribute to the development of the locality?

Biographical question E.g.: In the past, what was your job?

Example questions for the topic “Community participation in local planning”- How did the Commune Socio-Economic Development Plan (SEDP) in this year make?

Who participate in the plan? Which gender is more crowded, male or female?- Which activities in SEDP did you propose?- Which activities discussed in Village Development Plan (VDP) were not developed in

SEDP? - In your opinion, why were these activities not developed in SEDP?- What do you want to propose in accessing Community Development Fund (CDF) resources? - How does Commune Development Board (CDB) decide? - How are you satisfied with SEDP process?- What is your suggestion for change of SEDP in the next year?

3.4 Prepare the interview guideThe interview guide need include the following contents:

General information: purpose, subject and interview content; Themes, content frame and questions for each theme;

Page 74

Principles, requirements towards researchers when conducting interviews; Detail guide of how to open and close interview, how to question and encourage

interviewees to provide information.

3.5 Pre-test and finalize the questions or interview guide

Similarly to the quantitative survey, pre-test should be conducted to check the suitability and accuracy of the questions before finalizing the questions and interview guide. The pilot of key informant interview could be conducted with 02 – 03 key informants for each topic.

3.6 Conduct the interview

- Place to interview the informant: depending on the content, object to find an appropriate place for interview; avoid interviewing with the presence of a third person, especially presence of whom causing interference.

- Time to conduct interview: The interview should not take more than two hours (the most proper duration is between 01h-01h30’).

- Record the answer: It should consult the key informant about using record equipment. In the case of participants agree with the recording, it is still important to take note the content. Note to observe and record the changing emotions of the interviewee.

How to open a key informant interview? Confirm with the respondents of the anonymous of the interview. Explain why their

comments or observations on a certain topic are important for the study; Ask permission for recording the interview (if possible) and take notes during the

interview; Always be honest, straightforward and really concerned on what interviewees mention.

How to open a key informant interview? Confirm with the respondents of the anonymous of the interview. Explain why their

comments or observations on a certain topic are important for the study; Ask permission for recording the interview (if possible) and take notes during the

interview; Always be honest, straightforward and really concerned on what interviewees mention.

Let interviewees lead the interview Ensure that interviewees understand the question; Ensure that interviewees are able to provide all necessary information that the study needs; Ensure that they are willing to spend time and effort on the interview; Attempt to create a friendly communication, safety, sincerity and sympathy; But do not let the interview become a regular dialogue to avoid rambling, the point; Rule: Putting certain questions and then let interviewees free. Don’t interrupt them to

provide the information that they believe is important. Applying polling technique

Silence: keep quiet to wait for interviewee saying, can come with a nod and waiting eyes; Repeat: repeat the last sentence interviewee said object and ask them to continue; Motivation: encourage interviewee by expression of sympathy, or "yes", "right"; 

Page 75

Ask a long question: make interviewee answer more and more impression; Guide: do not drive interviewee’s reply by giving questions like "Do you think that ...".

It is better to ask, "What do you think about ..."; If the interviewee is talkative or digress from the topic: use tip to interrupt them but

does not make them displeased; Confirm: show that you have taken certain information about the topic of the interview

that makes the interviewee more open and less anxious for information disclosure. How to close the interview? 

Keep good relations with the interviewee for subsequent interviews; Be grateful attitude and appreciate the information that respondents had provided; Ask for contact information of respondents: personal phone number, email address (if

unavailable); Able to end the interview earlier than planned or requested for lengthening interviews

(if necessary). Tips to have a good key informant interview

Improve interview skill- Practice as much as possible;- Rehearse with others under supervision/help of an experienced person;- Rehearse interview with your interested topics with someone who are really expert in.

Recorder usage - Prepare a good recorder with battery;- Always try before interview.- Always note-taking for cases of unavailable recording

Some advices on the case that the key informants are women Before conducting the interviews, interviewers should understand family situation,

cultural and ethnic differences through village chiefs and local staffs to avoid the taboo and hurting interviewees, which possibly causes difficulties in collecting information.

Avoiding bothering when the women are busy. In the interview, the interviewers should avoid the presence of a third person that can

affect the key informants’ answers.3.7 Analyze and write reportSteps to analyze and write report:

After each interview, interviewer should prepare an interview summary in form of transcript, including the answers for each question, and assessment of interviewer about the attitude of interviewees;

Summering information by topics: for each topic, prepare a short summary (2-3 pages) to present most important findings, also to emphases on agreed opinions and opposite views;

Writing report: The report will analysis all information collected through key informant interviews. Depends on the purpose, each topic could be used to illustrate the findings from quantitative survey, to tell stories of best practices, or to develop to be a case study.

Page 76

Structure of a story with information for a key informant interview

Headline

Standfirst/Lead paragraphSummary

Photo Caption

Story (most important part)What happened? What did the project do? So what? What are the results?

ExamplesTestimonial/quotes form participants

Background (less important part)Details about the project, components, supporting data

ContactName(s) for further information, job title, address, phone (optional)

Page 77

Lecture 14: Focus group discussion1. Overview of focus group discussionWhat is a focus group discussion?Focus group discussion (also known as intensive group discussion, focus group interview; hereinafter referred to as “focus group”) aims to obtain qualitative information on attitude, perception, opinion and idea from a group of people who have in-depth understanding about specific topic. When use focus group?Group discussions are often used to assess the needs, collect opinions from many people to evaluate the same problem, or discuss on the controversial content, or collect new ideas. Advantages:

Provide a significant amount of information quickly and less expensive than individual interviews;

Very useful in understanding the concepts, attitudes and behavior of the community; Support for determining the appropriate questions for key informant interview.

Disadvantages: Facilitation of a focus group requires considerable skills; The flexible format makes it susceptible to facilitator bias, which can undermine the

validity and reliability of findings; Discussions can be sidetracked or dominated by a few vocal individuals; The information can be more difficult to analyze than ones from key informant

interview; The number of concerned issues in the focus group may be less than that ones of key

informant interview.2. Steps to facilitate a focus group discussion.In order to conduct the focus group discussion, six following steps should be undertaken: 2.1 Identify topics and information needed for each topica. Identify topicsThe number of topics in each focus group discussion depends on: (i) information needs collecting, and (ii) time and available sources.Example of topics for focus group discussion

- Women and men participation in local social and economic development planning;

- Market information accessibility, market linkages;- The effectiveness of vocational training and job consulting for women and men;- Advantages and disadvantages in the formation and development of interest groups.

b. Specify information needs for each topicIt is necessary to design appropriate questions; and twelve is the maximum number of questions for any group.Questions should be:

i. Short and to the point;ii. Focused on one dimension each;

Page 78

iii. Unambiguously worded;iv. Open-ended or sentence completion types;v. Non-threatening or embarrassing;

vi. Worded in a way that they cannot be answered with a simple “yes” or “no” answer (use “why” and “how” instead).

There are three types of focus group questions:(i) Engagement questions:

E.g.: Who in your household have attended the trainings of the project?(ii) Exploration questions

E.g.: Had the training topics / extension met your demand? Could you impact on selecting of the training topics / extension services?

(iii) Exit questionE.g.: Is there anything else you would like to say about why you didn’t access to the necessary production inputs?

2.2 Prepare guide for discussionThe discussion guide includes the following contents:

(1) General information: purpose, subject and discussion content;(2) Themes, content frame and questions for each theme;(3) Participants and number of participants;(4) Principles, requirements towards facilitator and assistant when conducting discussion;(5) Detail guide of how to open, facilitate and close the discussion; how to question and

encourage participants to talk; how to encourage women raise their voices if there are men in the discussion group

2.3 Select participantsThere are 8-12 participants participating in each discussion. Carefully consider the participants to take part in the FGD: participants from different classifications or only one class (e.g. both women and men or separate FGD for women and for men; both rich and poor households or only poor households, etc.) Focus groups participants can be recruited in anyone of a number of ways. Some of the most popular include:

- Nomination- Random selection- All members of the same group- Same role/job title- Volunteers

For example, for the topic "The participation of the community in designing local social and economic development plan", FGD could be organized for each of following group or mix of the groups:

- Women;- Men;- Poor households;- Rich households;- Ethnic minority households;

Page 79

- Kinh households.

If the group has both men and women, the facilitator should always remember to encourage women raise their voice in the discussionPre-test and revise the discussion guide • Pre-test is an important step in designing interview guide in general and designing

discussion in particular. • For group discussion, it is recommended to conduct a pre-test. • If it found any inappropriate content and discussion topics, it should be timely adjusted.

2.4 Conduct FGDIssues to be taken into account when carrying FGD:

Location: Depending on the topic and context of the FGD, participants to find the appropriate place.

Time: The ideal amount of time to set aside for a FGD is anywhere from 60 to 90 minutes. Discussion issues: Only when we start the discussion, the coordinator will give

specific details about the study to participants, to avoid the case of agreement among participants before the group discussions.

The responsibilities of facilitators: Directly instructing and facilitating the discussion; Raising the questions in a managed time; Encouraging the participants to take part in the discussion; Explaining all questions from the participants.

The responsibilities of assistants: Run a tape recorder during the session; Take notes; Note/record body language or other subtle but relevant clues; Allow the moderator to do all the talking during the group.

Focus Group IntroductionFOCUS GROUP INTRODUCTION

WELCOMEThanks for agreeing to be part of the focus group. We appreciate your willingness to participate.INTRODUCTIONSModerator; assistant moderatorPURPOSE OF FOCUS GROUPS

- We have been asked by _________________to conduct the focus groups.- The reason we are having these focus groups is to find out_______________.- We need your input and want you to share your honest and open thoughts with us.

GROUND RULES1. WE WANT YOU TO DO THE TALKING.

- We would like everyone to participate.- I may call on you if I haven't heard from you in a while.

2. THERE ARE NO RIGHT OR WRONG ANSWERS- Every person's experiences and opinions are important.

Page 80

- Speak up whether you agree or disagree.- We want to hear a wide range of opinions.

3. WHAT IS SAID IN THIS ROOM STAYS HERE- We want folks to feel comfortable sharing when sensitive issues come up.

4. WE WILL BE TAPE RECORDING THE GROUP- We want to capture everything you have to say.- We don't identify anyone by name in our report. You will remain anonymous.

Some suggestions to have a good FGD To get participant fully explain their answers:◦ “Can you talk about that more?”◦ “Help me understand what you mean”◦ “Can you give an example?”

Strategies to deal with challenging participants:◦ Self-appointed experts: “Thank you. What do other people think?”◦ The dominator: “Let’s have some other comments.”◦ The rambler: Stop eye contact; look at your watch; jump in at their inhale.◦ The shy participant: Make eye contact; call on them; smile at them.◦ The participant who talks very quietly: Ask them to repeat their response more loudly.

2.5 Analyze results and write report- Begin by transcribing all focus group tapes and inserting notes into transcribed material

where appropriate.- Clean up transcripts by stripping off nonessential words. Simultaneously assign each

participant comment/quote a separate line on the page as well as each new thought or idea therein. Label each line with the participant and group number.

- Each line is then entered into an Excel database. COMPILE

Use a separate Excel data base spreadsheet for each group. Within each spreadsheet, use one sheet per question. Label three columns on each sheet. One column for coding One column for the participant ID# One column for responses Enter each separate response or idea on a separate line with participant ID

attached. The coding column is filled in during the next phase - analysis ANALYZE

When all comments have been entered, look for common categories or themes across the entries for each question. The most ideal situation is to ask.

Once consensus has been achieved regarding the best categories for organizing the data, assign a number or letter to each category.

Then assign the number/letter of the category that best fits to each entry on the sheet. Use the Excel ‘Sort’ function to group entries by the categories you have assigned

to them.

Page 81

If some entries seem inconsistent for their category, consider re-categorizing or

adding another category. It may also be apparent that one or more categories can be collapsed.

Arrange categories from those with the largest number of entries to those with the smallest.

Repeat for each group. SYNTHESIZE

Identify category and sub-category heading titles. Write a short paragraph summarizing findings for each sub-category possibly

noting similarities and differences across groups. Add powerful quotes to each sub-section

An alternative to the spreadsheet methodMake a copy of the transcript after it has been cleaned and labeled. Working on a large table, cut entries into separate strips and run a glue stick over the back. Categorize by sticking entries onto separate sheets of paper labeled with broad headings. Re-categorize as indicated until you are satisfied with your groupings. Enter into a Word document.

Page 82