Rapid Appraisal Methods

8/17/2019 Rapid Appraisal Methods

1/6

PERFORMANCE MONITORING & EVALUATION

TIPSUSING RAPID APPRAISAL METHODS

ABOUT TIPSThese TIPS provide practical advice and suggestions to USAID managers on issues related to performancemonitoring and evaluation. This publication is a supplemental reference to the Automated DirectiveSystem (ADS) Chapter 203.

WHAT IS RAPID

APPRAISAL?

Rapid Appraisal (RA) is an approachthat draws on multiple evaluation

methods and techniques to quickly,yet systematically, collect data whentime in the field is limited. RApractices are also useful when thereare budget constraints or limitedavailability of reliable secondarydata. For example, time and budgetlimitations may preclude the optionof using representative samplesurveys.

BENEFITS – WHEN TO USE

RAPID APPRAISALMETHODS

Rapid appraisals are quick and canbe done at relatively low cost.Rapid appraisal methods can helpgather, analyze, and report relevantinformation for decision-makerswithin days or weeks. This is notpossible with sample surveys. RAscan be used in the following cases:

• for formative evaluations, to makemid-course corrections in projectdesign or implementation whencustomer or partner feedbackindicates a problem (See ADS203.3.6.1);

• when a key management decisionis required and there is inadequateinformation;

• for performance monitoring, whendata are collected and thetechniques are repeated over timefor measurement purposes;

• to better understand the issuesbehind performance monitoringdata; and

• for project pre-design assessment.

LIMITATIONS – WHEN

RAPID APPRAISALS ARE

NOT APPROPRIATE

Findings from rapid appraisals mayhave limited reliability and validity,and cannot be generalized to thelarger population. Accordingly,

rapid appraisal should not be thesole basis for summative or impactevaluations. Data can be biased andinaccurate unless multiple methodsare used to strengthen the validityof findings and careful preparation isundertaken prior to beginning fieldwork.

WHEN ARE RAPID

APPRAISAL

METHODSAPPROPRIATE?

Choosing between rapid appraisalmethods for an assessment or more

time-consuming methods, such assample surveys, should depend onbalancing several factors, listedbelow.

• Purpose of the study. Theimportance and nature of thedecision depending on it.

• Confidence in results. Theaccuracy, reliability, and validity of

NUMBER 5

2ND EDITION 2010

1

http://www.usaid.gov/policy/ads/200/203.pdfhttp://www.usaid.gov/policy/ads/200/203.pdfhttp://www.usaid.gov/policy/ads/200/203.pdfhttp://www.usaid.gov/policy/ads/200/203.pdf


2/6

findings needed for managementdecisions.

2

• Time frame. When a decisionmust be made.

• Resource constraints (budget).

• Evaluation questions to be

answered. (see TIPS 3: Preparingan Evaluation Statement of Work)

USE IN TYPES OFEVALUATION

Rapid appraisal methods are oftenused in formative evaluations.Findings are strengthened whenevaluators use triangulation(employing more than one datacollection method) as a check onthe validity of findings from any onemethod.

Rapid appraisal methods are alsoused in the context of summativeevaluations. The data from rapidappraisal methods and techniquescomplement the use of quantitativemethods such as surveys based onrepresentative sampling. Forexample, a randomized survey ofsmall holder farmers may tell you

that farmers have a difficult timeselling their goods at market, butmay not have provide you with thedetails of why this is occurring. Aresearcher could then useinterviews with farmers todetermine the details necessary toconstruct a more complete theoryof why it is difficult for small holderfarmers to sell their goods.

KEY PRINCIPLESFOR ENSURINGUSEFUL RAPID

APPRAISAL DATA

COLLECTION

No set of rules dictates whichmethods and techniques should beused in a given field situation;however, a number of key principles

can be followed to ensure thecollection of useful data in a rapidappraisal.

• Preparation is key. As in anyevaluation, the evaluation designand selection of methods mustbegin with a thorough

understanding of the evaluationquestions and the client’s needsfor evaluative information. Theclient’s intended uses of data mustguide the evaluation design andthe types of methods that areused.

• Triangulation increases the validityof findings. To lessen bias andstrengthen the validity of findingsfrom rapid appraisal methods and

techniques, it is imperative to usemultiple methods. In this way,data collected using one methodcan be compared to that collectedusing other methods, thus giving aresearcher the ability to generatevalid and reliable findings. If, forexample, data collected using KeyInformant Interviews reveal thesame findings as data collectedfrom Direct Observation andFocus Group Interviews, there is

less chance that the findings fromthe first method were due toresearcher bias or due to thefindings being outliers. Table 1summarizes common rapidappraisal methods and suggestshow findings from any onemethod can be strengthened bythe use of other methods.

COMMON RAPID

APPRAISALMETHODS

INTERVIEWS

This method involves one-on-oneinterviews with individuals or keyinformants selected for theirknowledge or diverse views.Interviews are qualitative, in-depthand semi-structured. Interviewguides are usually used and

questions may be further framedduring the interview, using subtleprobing techniques. Individualinterviews may be used to gaininformation on a general topic butcannot provide the in-depth insideknowledge on evaluation topics that

s

key informants may provide.

quickly.

MINISURVEYS

A minisurvey consists of interviews

with between five to fifty individuals,usually selected using non-probability sampling (sampling inwhich respondents are chosen basedon their understanding of issuesrelated to a purpose or specificquestions, usually used when samplesizes are small and time or access toareas is limited). Structuredquestionnaires are used with alimited number of close-endedquestions. Minisurveys generate

quantitative data that can often becollected and analyzed

FOCUS GROUPS

The focus group is a gathering of ahomogeneous body of five to twelveparticipants to discuss issues andexperiences among themselves.These are used to test an idea or toget a reaction on specific topics. Amoderator introduces the topic,timulates and focuses the

EVALUATION METHODS

COMMONLY USED IN RAPID

APPRAISAL

• Interviews

• Community Discussions

• Exit Polling

• Transect Walks (see p. 3)

• Focus Groups

• Minisurveys

• Community Mapping

• Secondary Data Collection

• Group Discussions

• Customer Service Surveys

• Direct Observation


3/6

COMMUNITY DISCUSSIONS

3

documents theconversation.

responddirectly to the moderator. community discussions. The

discussion, and prevents dominationof discussion by a few, while anotherevaluator

This method takes place at a publicmeeting that is open to allcommunity members; it can besuccessfully moderated with asmany as 100 or more people. Theprimary interaction is between theparticipants while the moderatorleads the discussion and asksquestions following a carefullyprepared interview guide.

GROUP DISCUSSIONS

This method involves the selectionof approximately five participantswho are knowledgeable about agiven topic and are comfortableenough with one another to freelydiscuss the issue as a group. Themoderator introduces the topic andkeeps the discussion going whileanother evaluator records thediscussion. Participants talk amongeach other rather than

DIRECT OBSERVATION

Teams of observers record whatthey hear and see at a program siteusing a detailed observation form.Observation may be of the physicalsurrounding or of ongoing activities,processes, or interactions.

COLLECTING SECONDARY

DATA

This method involves the on-sitecollection of existing secondarydata, such as export sales, loaninformation, health service statistics,etc. These data are an importantaugmentation to informationcollected using qualitative methodssuch as interviews, focus groups, and

evaluator must be able to quicklydetermine the validity and reliabilityof the data. (see TIPS 12: Indicatorand Data Quality)

TRANSECT WALKS

rticipatory

COMMUNITY MAPPING

nique

LOGYTHE ROLE OF TECHNO

IN RAPID APPRAISAL

Certain equipment and technologies

can aid the rapid collection of dataand help to decrease the incidence oferrors. These include, for example,hand held computers or personaldigital assistants (PDAs) for datainput, cellular phones, digitalrecording devices for interviews,videotaping and photography, and theuse of geographic information syste

The transect walk is a paapproach in which the evaluator

asks a selected community memberto walk with him or her, forexample, through the center oftown, from one end of a village tothe other, or through a market.The evaluator asks the individual,usually a key informant, to point outand discuss important sites,neighborhoods, businesses, etc., andto discuss related issues.

ms(GIS) data and aerial photographs.

Community mapping is a techthat requires the participation ofresidents on a program site. It canbe used to help locate naturalresources, routes, service deliverypoints, regional markets, troublespots, etc., on a map of the area, orto use residents’ feedback to drivethe development of a map thatincludes such information.


4/6

COMMON RAPID APPRAISAL METHODS

Table 1

MethodUseful for

ProvidingExample Advantages Limitations

Further

References

INDIVIDUAL INTERVIEWS

Interviews − A general overview ofthe topic fromsomeone who has a

broad knowledge andin-depth experienceand understanding(key informant) or in-depth information ona very specific topic orsubtopic (individual)

− Suggestions andrecommendations toimprove key aspectsof a program

Key informant:Interview withprogram

implementationdirector

Interview withdirector of a regionaltrade association

Individual:Interview with anactivity manager withinan overalldevelopment program

Interview with a localentrepreneur trying toenter export trade

− Provides in-depth,inside information on specific issues

from theindividualsperspective andexperience

− Flexibility permitsexploringunanticipatedtopics

− Easy to administer

− Low cost

− Susceptible tointerviewer andselection biases

− Individual interviews lack thebroaderunderstanding andinsight that a keyinformant canprovide

TIPS No. 2,Conducting KeyInformant Interviews

K. Kumar, ConductingKey Informant Surveysin DevelopingCountries, 1986

Bamberger, Rugh, andMabry, Real WorldEvaluation, 2006

UNICEF Website: M&ETraining Modules:Overview of RAPTechniques

Minisurveys − Quantitative data onnarrowly focusedquestions, for arelativelyhomogeneouspopulation, whenrepresentativesampling is notpossible or required

− Quick data onattitudes, beliefs,behaviors ofbeneficiaries orpartners

− A customer serviceassessment

− Rapid exit interviewsafter voting

− Quantitative datafrom multiplerespondents

− Low cost

− Findings are lessgeneralizable thanthose from samplesurveys unless theuniverse of thepopulation issurveyed

TIPS No. 9,Conducting CustomerService Assessments

K. Kumar, ConductingMini Surveys inDeveloping Countries,1990

Bamberger, Rugh, andMabry, RealWorld

Evaluation, 2006 onpurposeful sampling

GROUP INTERVIEWS

Focus Groups − Customer views onservices, products,benefits

− Information on

implementationproblems

− Suggestions andrecommendations forimproving specificactivities

− Discussion onexperience relatedto a specific programintervention

− Effects of a newbusiness regulationor proposed pricechanges

− Group discussionmay reduceinhibitions,allowing freeexchange of ideas

− Low cost

− Discussion may bedominated by afew individualsunless the processis facilitated/managed well

TIPS No. 10,Conducting FocusGroup Interviews

K. Kumar, Conducting

Group Interviews inDeveloping Countries,1987

T. Greenbaum,Moderating FocusGroups: A PracticalGuide for GroupFacilitation, 2000

4

http://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.doc


5/6

Group

Discussions

− Understanding ofissues from differentperspectives andexperiences ofparticipants from aspecific subpopulation

− Discussion withyoung women onaccess to prenataland infant care

− Discussion withentrepreneurs aboutexport regulations

− Small group sizeallows fullparticipation

− Allows goodunderstanding ofspecific topics

− Low cost

− Findings cannot begeneralized to alarger population

Bamberger, Rugh, andMabry, RealWorldEvaluation, 2006

UNICEF Website: M&ETraining Modules:Community Meetings

Community

Discussions

− Understanding of anissue or topic from awide range ofparticipants from keyevaluation sites withina village, town, city, orcity neighborhood

− A Town Hallmeeting

− Yields a widerange of opinionson issuesimportant toparticipants

− A great deal ofinformation can beobtained at onepoint of time

− Findings cannot begeneralized tolarger populationor tosubpopulations ofconcern

− Larger groupsdifficult tomoderate

Bamberger, Rugh, andMabry, RealWorldEvaluation, 2006

UNICEF Website: M&ETraining Modules:Community Meetings

ADDITIONAL COMMONLY USED TECHNIQUES

Direct

Observation

− Visual data on physicalinfrastructure,supplies, conditions

− Information about anagency’s or business’sdelivery systems,services

− Insights into behaviorsor events

− Market place toobserve goods beingbought and sold,who is involved,

sales interactions

− Confirms datafrom interviews

− Low cost

− Observer biasunless two tothree evaluatorsobserve same

place or activity

TIPS No. 4, UsingDirect ObservationTechniques

WFP Website:

Monitoring & EvaluationGuidelines: What IsDirect Observation andWhen Should It Be Used?

Collecting

Secondary

Data

− Validity to findingsgathered frominterviews and groupdiscussions

− Microenterprisebank loan info.

− Value and volume ofexports

− Number of peopleserved by a healthclinic, social serviceprovider

− Quick, low costway of obtainingimportantquantitative data

− Must be able todeterminereliability andvalidity of data

TIPS No. 12,Guidelines forIndicator and DataQuality

PARTICIPATORY TECHNIQUES

Transect

Walks

− Important visual andlocational informationand a deeperunderstanding ofsituations and issues

− Walk with keyinformant from oneend of a village orurban neighborhoodto another, througha market place, etc.

− Insiders viewpoint

− Quick way to findout location ofplaces of interestto the evaluator

− Low cost

− Susceptible tointerviewer andselection biases



CommunityMapping

− Info. on locationsimportant for datacollection that couldbe difficult to find

− Quick comprehensionon spatial location ofservices/resources in aregion which can giveinsight to access issues

− Map of village andsurrounding areawith locations ofmarkets, water andfuel sources, conflictareas, etc.

− Importantlocational datawhen there are nodetailed maps ofthe program site

− Rough locationalinformation



5

http://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-8_overview-RAP-techniques.dochttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_13.pdfhttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.dochttp://www.ceecis.org/remf/Service3/unicef_eng/module5/docs/5-2-4_interviews-community-meetings.doc


6/6

References Cited

M. Bamberger, J. Rugh, and L. Mabry, Real World Evaluation. Working Under Budget, Time, Data, and PoliticalConstraints. Sage Publications, Thousand Oaks, CA, 2006.

T. Greenbaum, Moderating Focus Groups: A Practical Guide for Group Facilitation. Sage Publications, Thousand Oaks,CA, 2000.

K. Kumar, “Conducting Mini Surveys in Developing Countries,” USAID Program Design and Evaluation Methodology

Report No. 15, 1990 (revised 2006).

K. Kumar, “Conducting Group Interviews in Developing Countries,” USAID Program Design and Evaluation Methodology Report No. 8, 1987.

K. Kumar, “Conducting Key Informant Interviews in Developing Countries,” USAID Program Design and Evaluation Methodology Report No. 13, 1989.

For more information:TIPS publications are available online at [insert website].

Acknowledgements:

Our thanks to those whose experience and insights helped shape this publication including USAID’s Office ofManagement Policy, Budget and Performance (MPBP). This publication was authored by Patricia Vondal, PhD., ofManagement Systems International.

Comments regarding this publication can be directed to:

Gerald Britan, Ph.D.Tel: (202) [email protected]

Contracted under RAN-M-00-04-00049-A-FY0S-84Integrated Managing for Results II

6
mailto:[email protected]:[email protected]

Rapid Appraisal Methods

Documents

Transcript of Rapid Appraisal Methods