Qualitative and Qualitative and Quantitative Poverty Quantitative Poverty

Appraisal:Appraisal:

Maximizing Maximizing Complementarities, Complementarities, Minimizing TradeoffsMinimizing Tradeoffs

Chris BarrettChris BarrettCornell UniversityCornell UniversityAugust 16, 2003August 16, 2003

IAAE Learning Workshop IAAE Learning Workshop Durban, South AfricaDurban, South Africa

Good background resources:Good background resources:

- Ravi Kanbur, ed.,- Ravi Kanbur, ed., Q-Squared: Q-Squared: Combining Qualitative and Combining Qualitative and Quantitative Methods In Poverty Quantitative Methods In Poverty AppraisalAppraisal (Delhi: Permanent Black, (Delhi: Permanent Black, 2003)2003)

- Biju Rao and Michael Woolcock - Biju Rao and Michael Woolcock chapter in forthcoming IMF/World chapter in forthcoming IMF/World Bank Tool Kit on Evaluation MethodsBank Tool Kit on Evaluation Methods

Significant recent progress in Significant recent progress in both qualitative (QUAL) and both qualitative (QUAL) and quantitative (QUANT) methods:quantitative (QUANT) methods:

- rapid rise of participatory - rapid rise of participatory appraisal (PA) appraisal (PA)

methodsmethods- emergence of widespread, - emergence of widespread,

nationally representative nationally representative household survey household survey

data, data, including including panels.panels.

Are QUAL and QUANT Are QUAL and QUANT complements or complements or substitutes?substitutes?

There is considerable conflict There is considerable conflict among the practitioners of among the practitioners of each … does that mean the each … does that mean the methods necessarily conflict methods necessarily conflict too???too???

Importance of being self-Importance of being self-reflexive and self-criticalreflexive and self-critical

Dimensions of QUAL-QUANT Dimensions of QUAL-QUANT Difference:Difference: Be clear about focus of question:Be clear about focus of question:

(1) Data collection methods(1) Data collection methods

(2) Data types(2) Data types (3) Data analysis methods(3) Data analysis methods

(4) Audience(4) Audience

Qual-Quant DimensionsQual-Quant Dimensions

Data collection methodsData collection methods A

nalytical Coverage

General

Specific

Passive Active Population Involvement in Research

Census

Autobiography

PRA

Random Sample Surveys

Qual-Quant DimensionsQual-Quant Dimensions

““Qualitative”Qualitative” “Quantitative”“Quantitative”

Data types:Data types:CategoricalCategorical Ordinal Ordinal Cardinal Cardinal

Each data collection Each data collection methodmethod can yield both can yield both non-numerical and numerical non-numerical and numerical data typesdata types

Qual-Quant DimensionsQual-Quant Dimensions

““Qualitative”Qualitative” “Quantitative”“Quantitative”

Data analysis methods:Data analysis methods:InductiveInductive Deductive Deductive

Related to the specific-general data Related to the specific-general data collection methods distinction, there’s collection methods distinction, there’s often (not always) a difference in often (not always) a difference in analysis methods.analysis methods.

Qual-Quant DimensionsQual-Quant DimensionsAudience:Audience:Local communityLocal community Global/national Global/national

policymakerspolicymakers

QUAL researchers often worry out loud QUAL researchers often worry out loud about local empowerment and the intrinsic about local empowerment and the intrinsic value of the research process. QUANT value of the research process. QUANT types tend to worry about “big picture” or types tend to worry about “big picture” or “take home” messages, about “speaking “take home” messages, about “speaking truth to power”truth to power”

Key advantages of QUAL Key advantages of QUAL approaches:approaches:

Allow more immediate probing in response to Allow more immediate probing in response to unanticipated results (adaptability)unanticipated results (adaptability)

More nuanced and textured for complex, More nuanced and textured for complex, unmeasurable concepts (e.g., power, opportunity, unmeasurable concepts (e.g., power, opportunity, security)security)

Let subjects speak for themselvesLet subjects speak for themselves

More oriented toward uncovering More oriented toward uncovering processes/mecahnismsprocesses/mecahnisms

Key advantages of QUANT Key advantages of QUANT approaches:approaches:

Use of sampling frames and randomization reduces Use of sampling frames and randomization reduces inferential bias: coincidence and causalityinferential bias: coincidence and causality

Uniformity/structure in design/definitions fosters Uniformity/structure in design/definitions fosters replicability over time (longitudinal analysis) and replicability over time (longitudinal analysis) and across samples (comparative analysis)across samples (comparative analysis)

Far easier aggregability – few scaling up problemsFar easier aggregability – few scaling up problems

Myths about QUAL-QUANT Myths about QUAL-QUANT differencesdifferences

(1)(1) One more/less extractive than the other (“ethical One more/less extractive than the other (“ethical superiority”)superiority”)

(2)(2) One more/less contextual than the other (“historical One more/less contextual than the other (“historical superiority”)superiority”)

(3)(3) One inherently numerical/non-numericalOne inherently numerical/non-numerical(“statistical superiority”)(“statistical superiority”)

(4) One more “rigorous” than the other(4) One more “rigorous” than the other(“scientific superiority”)(“scientific superiority”)

Bad practice is bad practice, whatever the method...Bad practice is bad practice, whatever the method...

Key questionKey question: when and how is good practice within : when and how is good practice within one strand still wanting? How can the other fill the one strand still wanting? How can the other fill the blanks?blanks?

Mixing methodsMixing methods

When brought together, QUAL and QUANT When brought together, QUAL and QUANT rarely have similar status, especially in rarely have similar status, especially in policy discourse, where aggregability and policy discourse, where aggregability and the “illusion of precision” commonly the “illusion of precision” commonly dominate. dominate.

Improve analysis by mixing the two … taking Improve analysis by mixing the two … taking the “con” out of econometrics, generalizing the “con” out of econometrics, generalizing beyond the “part” of participatory methodsbeyond the “part” of participatory methods

Why mix methods?Why mix methods?(1)(1) QUAL can improve QUANT by:QUAL can improve QUANT by:

- Improving survey/instrument design. Data are Improving survey/instrument design. Data are social products, so need to understand sourcesocial products, so need to understand source

- Improving specification of formal modelsImproving specification of formal models- Improving statistical inference Improving statistical inference - Identifying suitable instrumental variables, Identifying suitable instrumental variables,

exclusionary restrictions, etc.exclusionary restrictions, etc.- Shedding light on outliers (“It helps to have Shedding light on outliers (“It helps to have

had tea with an outlier” – Biju Rao)had tea with an outlier” – Biju Rao)- Highlighting likely sources of measurement Highlighting likely sources of measurement

error (the “Chai stall” error – Ron Herring)error (the “Chai stall” error – Ron Herring)- Breathing life into otherwise abstract numbersBreathing life into otherwise abstract numbers

Why mix methods?Why mix methods?(1)(1)QUANT can improve QUAL by:QUANT can improve QUAL by:

- Reducing researcher-induced bias Reducing researcher-induced bias by structure and replicabilityby structure and replicability

- Facilitating comparabilityFacilitating comparability- Facilitating aggregabilityFacilitating aggregability- Broadening the audience for resultsBroadening the audience for results- Fostering more precise criteria for Fostering more precise criteria for

demonstrating causal relationshipsdemonstrating causal relationships

Different methods of mixingDifferent methods of mixing

““Sequential mixing” or “classical integration”Sequential mixing” or “classical integration”

Practitioners of each method do their best Practitioners of each method do their best with their own tools on same problem, with their own tools on same problem, sometimes taking outputs from one as sometimes taking outputs from one as intermediate inputs to another. Then intermediate inputs to another. Then triangulate to get an integrated result.triangulate to get an integrated result.

Sequential mixingSequential mixingExample: Understanding welfare transitionsExample: Understanding welfare transitions

Step 1: Panel survey data collection to construct transition matrices Step 1: Panel survey data collection to construct transition matrices and change measures.and change measures. Poor Poort t NonpoorNonpoortt

PoorPoort+1t+1

Step 2: Draw several households Step 2: Draw several households from each of 6 cells in matrix Nonpoorfrom each of 6 cells in matrix Nonpoort+1t+1

and do detailed oral histories.and do detailed oral histories.

Why? Capture omitted variables, check transitions, 2Why? Capture omitted variables, check transitions, 2ndnd method of method of inference, problem of identifying thresholds econometrically, value inference, problem of identifying thresholds econometrically, value of stories for policy audiences.of stories for policy audiences.

__ ++

__

++ __ ++

Different methods of mixingDifferent methods of mixing

““Simultaneous mixing” or “Bayesian integration”Simultaneous mixing” or “Bayesian integration”Iterative approach to using one method to inform another, then Iterative approach to using one method to inform another, then

back to the first, etc., keeping multiple methods interactive back to the first, etc., keeping multiple methods interactive throughout the research process to update researchers’ priors throughout the research process to update researchers’ priors continuously. continuously.

Feedback loop yields a homeostatic research mechanism:Feedback loop yields a homeostatic research mechanism:““ethnography” precedes “participatory” which in turn precedes ethnography” precedes “participatory” which in turn precedes “survey” in dictionary … and the in field, too!“survey” in dictionary … and the in field, too!

Ongoing, creative tension between methods helps ensure Ongoing, creative tension between methods helps ensure originality, robustness and relevance of resultsoriginality, robustness and relevance of results

Different methods of mixingDifferent methods of mixingExample:Example: Pastoral Risk Management (PARIMA) Pastoral Risk Management (PARIMA)

project based on multidisiplinary integration:project based on multidisiplinary integration:(a) What does it mean to poor or vulnerable in this (a) What does it mean to poor or vulnerable in this setting? How does this vary across individuals, setting? How does this vary across individuals, households, communities and time? [asking the households, communities and time? [asking the right questions or the right people at right time?]right questions or the right people at right time?]

(b) Derivative from (a), are we measuring the (b) Derivative from (a), are we measuring the correct variables and in the right manner?correct variables and in the right manner?

(c) Is our inference consistent (i) across methods (c) Is our inference consistent (i) across methods (a test of robustness) and (i) with local (a test of robustness) and (i) with local understandings of the problem(s) (a test of understandings of the problem(s) (a test of relevance)?relevance)?

Tools developed/employedTools developed/employed- Participatory risk mapping (Smith et al. WD - Participatory risk mapping (Smith et al. WD 2000, JDS 2001) to identify relevant threats (e.g., 2000, JDS 2001) to identify relevant threats (e.g., human health) open-ended, spatially-explicit, human health) open-ended, spatially-explicit, pseudo-cardinalpseudo-cardinal- Quarterly repeated surveys with open-ended - Quarterly repeated surveys with open-ended sections and mixed modules: sections and mixed modules: (i) complex property rights; climate forecasting, (i) complex property rights; climate forecasting, resource conflict; land use history; livelihoods resource conflict; land use history; livelihoods strategies, etc.strategies, etc.(ii) complementarity at multiples levels of analysis (ii) complementarity at multiples levels of analysis and different methods (e.g., livestock marketing and different methods (e.g., livestock marketing with data from households, markets and traders)with data from households, markets and traders)

Example:Example: ParticipatorParticipatory risk maps y risk maps of rainfall of rainfall and and drought risk drought risk (Smith et (Smith et al. 2001 al. 2001 JDSJDS))

Walking On Both LegsWalking On Both LegsDevelopment scholars and practitioners increasingly Development scholars and practitioners increasingly recognize the complementarity between qualitative recognize the complementarity between qualitative and quantitative methods. and quantitative methods.

There are big gains to be enjoyed from relatively There are big gains to be enjoyed from relatively small movements along the QUAL-QUANT axes in small movements along the QUAL-QUANT axes in any of several dimensions.any of several dimensions.

Tradeoffs grow, however, so multidisciplinary mixing Tradeoffs grow, however, so multidisciplinary mixing seems best, whether sequential or simultaneous, to seems best, whether sequential or simultaneous, to take advantage of inherent complementarities from take advantage of inherent complementarities from diversity of methods.diversity of methods.

Walking On Both LegsWalking On Both LegsBut much remains to be done …But much remains to be done …

Need work on Need work on (i) vocabulary(i) vocabulary(ii) field methods(ii) field methods(iii) data cross-referencing(iii) data cross-referencing(iv) fostering respectful dialogue(iv) fostering respectful dialogue