5 May 2009 @ Crawford School 1 Comparative Case Studies – 1 Semester 1, 2009 POGO8096/8196:...

5 May 2009 @ Crawford School 1

Comparative Comparative Case Studies – 1Case Studies – 1

Semester 1, 2009Semester 1, 2009

POGO8096/8196: Research POGO8096/8196: Research MethodsMethods

Crawford School of Economics and Government Crawford School of Economics and Government

25 May 2009 @ Crawford School

This weekThis week

Intentional selection of observationsIntentional selection of observations How to choose observations?How to choose observations? How to avoid problematic causal How to avoid problematic causal

inference?inference? Some controversiesSome controversies

Observations vs. casesObservations vs. cases Objectives of qualitative researchObjectives of qualitative research


Causal thinking: ReviewCausal thinking: Review

A hypothesis: A hypothesis: XX causes causes YY XX = a = a keykey independent variable independent variable YY = a dependent variable = a dependent variable

To claim that To claim that XX causes causes YY, … , … The effects of the “other” variables (The effects of the “other” variables (Z)Z)

should be controlled. should be controlled. XX and and ZZ should be causally prior to should be causally prior to YY. . (And, of course, (And, of course, XX and and YY must be must be

correlated.) correlated.)


True experimental designTrue experimental design An ideal design but applicability is limited.An ideal design but applicability is limited.

Quasi-experimental/correlational Quasi-experimental/correlational designsdesigns Both designs are commonly used not only Both designs are commonly used not only

in quantitative studies but also in in quantitative studies but also in qualitative studies.qualitative studies.

Design without a control groupDesign without a control group = a design without a variation in = a design without a variation in XX..

Research designs: Research designs: ReviewReview


Controlling other factorsControlling other factors

Then, how to control other factors in Then, how to control other factors in studies other than true experiments. studies other than true experiments. Quantitative researchers collect a large Quantitative researchers collect a large

number of observations with more than one number of observations with more than one independent variable and, then, apply a independent variable and, then, apply a technique of “multivariate statistical technique of “multivariate statistical analysis.” analysis.”

Qualitative researchers conduct “intentional Qualitative researchers conduct “intentional selection of observations” – data collection selection of observations” – data collection and analysis proceeding together. and analysis proceeding together.


Intentional selection Intentional selection of observationsof observations

The “best” intentional selection of The “best” intentional selection of observations:observations:

1.1. Choose observations so that other Choose observations so that other factors (factors (ZZ) are as similar as possible – ) are as similar as possible – to avoid “omitted variable bias”.to avoid “omitted variable bias”.

2.2. Choose Choose XX and and ZZ that are causally prior that are causally prior to to Y Y –– to avoid “simultaneous bias”. to avoid “simultaneous bias”.

3.3. You need to ensure variation in the You need to ensure variation in the key independent variable (key independent variable (XX) ) without without regard to the dependent variableregard to the dependent variable ( (YY) – ) – to avoid “selection bias”.to avoid “selection bias”.


An exampleAn example

Y = The level of democracy in an African Y = The level of democracy in an African nationnation

X = American “aid to democracy” in AfricaX = American “aid to democracy” in Africa

XX ZZ11 ZZ22

Design Design 11

Country Country AA

High High AidAid

Country Country BB

Low AidLow Aid

Design Design 22

Country Country AA

High High AidAid

High High Edu.Edu.

PeacefuPeacefull

Country Country BB

Low AidLow Aid High High Edu.Edu.

PeacefuPeacefull


Data collection stagesData collection stages

Ideally, two stages of data collection.Ideally, two stages of data collection.1.1. Given prior knowledge (i.e., implications Given prior knowledge (i.e., implications

from theories, previous empirical findings, from theories, previous empirical findings, common sense, your own experiences, etc), common sense, your own experiences, etc), collect information about collect information about independentindependent variables and decide appropriate variables and decide appropriate observations for hypothesis testing.observations for hypothesis testing.

2.2. Collect information about your Collect information about your dependentdependent variable and compare observations. variable and compare observations.


Problems – 1Problems – 1

The difference between observations may The difference between observations may be due to some significant “omitted be due to some significant “omitted variables.”variables.” Why is Country B not getting aid from the US, Why is Country B not getting aid from the US,

if it has as favorable conditions as Country A?if it has as favorable conditions as Country A? It is difficult, if not impossible, to find a It is difficult, if not impossible, to find a

proper set of observations.proper set of observations. Can we find two African countries that are Can we find two African countries that are

similar in many aspects but only one of them similar in many aspects but only one of them receiving aid from the US?receiving aid from the US?


Problems – 2Problems – 2

The control by intentional selection of The control by intentional selection of observations can improve the likelihood observations can improve the likelihood of obtaining valid causal inferences, but of obtaining valid causal inferences, but we should acknowledge its limitations. we should acknowledge its limitations. Practically, you can only control a few Practically, you can only control a few

variables in qualitative comparative variables in qualitative comparative studies. studies.

It is thus very important to think which It is thus very important to think which of many potential “third factors” must of many potential “third factors” must be controlled. be controlled.


What to control? – 1What to control? – 1

What should be controlled? What should be controlled? Without thinking causal Without thinking causal

theories/hypotheses and reviewing theories/hypotheses and reviewing extant empirical findings, you cannot extant empirical findings, you cannot decide which independent variables decide which independent variables should be included or excluded in your should be included or excluded in your analysis.analysis.

Logical thinking (about causality) and Logical thinking (about causality) and careful review of existing studies are careful review of existing studies are very important!very important!


What to control? – 2What to control? – 2

More specifically,More specifically, You should control You should control ZZ if you expect that if you expect that

ZZ is correlated with BOTH is correlated with BOTH XX and and YY.. If you expect that If you expect that ZZ is correlated with is correlated with XX (or (or

YY) but not with ) but not with YY (or (or XX), omitting Z does not ), omitting Z does not cause omitted variable bias, while including cause omitted variable bias, while including ZZ causes inefficiency. causes inefficiency.

You should NOT control You should NOT control ZZ that that happened after happened after XX and and YY..


““Selection bias”Selection bias”

Another potential problem very common Another potential problem very common in qualitative comparative studies is a in qualitative comparative studies is a “selection bias” problem.“selection bias” problem.

A typical selection bias problem:A typical selection bias problem: We know what we want to see as the We know what we want to see as the

outcome of the research (i.e., the outcome of the research (i.e., the confirmation of a favorable hypothesis). confirmation of a favorable hypothesis). Then, intentionally or unintentionally, we Then, intentionally or unintentionally, we select observations on the basis of select observations on the basis of combinations of combinations of XX and and YY that support the that support the desirable conclusion. desirable conclusion.


Rules of selection – 1Rules of selection – 1

How to avoid selection bias?How to avoid selection bias? General rulesGeneral rules

The best way to avoid selection bias is to The best way to avoid selection bias is to use a “random sampling” of (many) use a “random sampling” of (many) observations, but it is not always applicable.observations, but it is not always applicable.

If If intentionalintentional selection is required, we selection is required, we should attempt to get these observations should attempt to get these observations which are pivotal in deciding the question which are pivotal in deciding the question of interest, not those which merely support of interest, not those which merely support our position. our position.


Rules of selection – 2Rules of selection – 2

Specific rulesSpecific rules If there is no variation in the key If there is no variation in the key

independent variable, you cannot test your independent variable, you cannot test your hypothesis (i.e., a research design without a hypothesis (i.e., a research design without a control group.) control group.)

Selection of observations based on the Selection of observations based on the values of the values of the dependentdependent variable causes variable causes biased inference.biased inference.

Selection of observations based on the Selection of observations based on the values of the values of the independentindependent variable/s causes variable/s causes no biased inference. no biased inference.


Bad examplesBad examples

Inkeles and Rossi (1956) compared a numInkeles and Rossi (1956) compared a number of industrialized nations and argued thber of industrialized nations and argued that industrialization leads to the particular pat industrialization leads to the particular prestige hierarchy of occupations. restige hierarchy of occupations.

A research claiming that authoritarian rule lA research claiming that authoritarian rule leads to high economic growth rates, based eads to high economic growth rates, based on authoritarian observations with high groon authoritarian observations with high growth rates and non-authoritarian observatiowth rates and non-authoritarian observations with low growth rates.ns with low growth rates.


Potential pitfalls: Potential pitfalls: SummarySummary

You fail to control the You fail to control the effects of other effects of other significantly relevant significantly relevant variables. variables.

“ “omitted variable omitted variable bias” (i.e., problem bias” (i.e., problem associated with associated with selection of selection of variablesvariables))

You do not select You do not select appropriate appropriate observations.observations.

“ “selection bias” selection bias” (i.e., problem (i.e., problem associated with associated with selection of selection of observationsobservations))

Your key Your key independent variable independent variable is partly explained by is partly explained by your dependent your dependent variable. variable.

“ “simultaneous simultaneous bias” (an advanced bias” (an advanced topic not covered in topic not covered in this course).this course).


RemarkRemark

These problems are These problems are notnot specific to specific to qualitative comparative studies.qualitative comparative studies. In quantitative/statistical studies, particularly In quantitative/statistical studies, particularly

ones with the random sampling of observations, ones with the random sampling of observations, the degree of selection bias is small.the degree of selection bias is small.

Except in true-experimental studies, omitted Except in true-experimental studies, omitted variable bias is always present to a greater or variable bias is always present to a greater or lesser degree.lesser degree.

In correlational studies, particularly ones with In correlational studies, particularly ones with survey data, simultaneous bias is common. survey data, simultaneous bias is common.


ControversiesControversies

Is a single-case study valid in Is a single-case study valid in making causal inference?making causal inference?

Do we need to make causal Do we need to make causal inference in qualitative research? inference in qualitative research?


Observations vs. casesObservations vs. cases An observationAn observation

… … is defined as an is defined as an observedobserved answer (to a answer (to a question), behavior, or phenomenon for one question), behavior, or phenomenon for one unit for one dependent variable. It includes unit for one dependent variable. It includes information on the values of the independent information on the values of the independent variables.variables.

= the unit of analysis= the unit of analysis A caseA case

… … can refer to a single unit, on which many can refer to a single unit, on which many variables are measured (= an observation).variables are measured (= an observation).

… … can also refer even to a large domain for can also refer even to a large domain for analysis.analysis.


Example – 1Example – 1

IndividualIndividualss

Q1 (DV)Q1 (DV) Q2 (IV-1)Q2 (IV-1) Q3 (IV-2)Q3 (IV-2)

Mr. AMr. A YesYes HighHigh AgreeAgree

Mr. BMr. B NoNo MediumMedium DisagreeDisagree

Mrs. CMrs. C YesYes LowLow DisagreeDisagree

Mrs. DMrs. D YesYes HighHigh AgreeAgree

• 4 observations and 3 variables

• Observations = individuals


Example – 2Example – 2

A comparative study of modern wars.A comparative study of modern wars. Observations = modern warsObservations = modern wars World War II is one of the “observations.”World War II is one of the “observations.” The number of battle deaths may be a dependent The number of battle deaths may be a dependent

variable. variable. A comparative study of battles during WW IIA comparative study of battles during WW II

Observations = battles during the WW II. Observations = battles during the WW II. World War II is a “case” that includes a large World War II is a “case” that includes a large

number of observations.number of observations. The number of battle deaths may be a dependent The number of battle deaths may be a dependent

variable. variable.


A single case study?A single case study? A single case study? A single case study?

A single “case” study is valid as long as there A single “case” study is valid as long as there are more than one observation given that are more than one observation given that case. case.

A single “observation” study is totally invalid A single “observation” study is totally invalid to make causal inference. (No variation in to make causal inference. (No variation in XX!)!)

What to choose? What to choose? You should choose “appropriate” You should choose “appropriate”

observations for your comparative study.observations for your comparative study. Even in a single case study, you can increase Even in a single case study, you can increase

the number of observations. the number of observations.


Qualitative researchQualitative research

Manheim, Rich, Willnat (p. 314-319)Manheim, Rich, Willnat (p. 314-319) Qualitative researchers are less likely to be intQualitative researchers are less likely to be int

erested in erested in testingtesting a pre-formed hypothesis. a pre-formed hypothesis. Qualitative researchers are usually unconcerneQualitative researchers are usually unconcerne

d with “holding constant” some factors in order d with “holding constant” some factors in order to make causal inference.to make causal inference.

Qualitative researchers are often far less concQualitative researchers are often far less concerned with observing “representative” cases.erned with observing “representative” cases.


Objectives – 1Objectives – 1

Is it impossible/meaningless to make Is it impossible/meaningless to make causal inference in qualitative research? causal inference in qualitative research? No. As long as you have a hypothesis that No. As long as you have a hypothesis that

you want to test, you should design an you want to test, you should design an effective comparative research to make effective comparative research to make causal inference.causal inference.

Do we always have to make causal Do we always have to make causal inference?inference? No. There are multiple objectives to collect No. There are multiple objectives to collect

qualitative data. qualitative data.


Objectives – 2Objectives – 2 Often, a question can be better answered Often, a question can be better answered

by “mixing” various research methods and by “mixing” various research methods and collecting various types of empirical collecting various types of empirical information.information. Preliminary (qualitative) data collection that Preliminary (qualitative) data collection that

helps you to formulate your hypothesis. helps you to formulate your hypothesis. Qualitative data collection (even a single-Qualitative data collection (even a single-

observation study) to understand why statistical observation study) to understand why statistical analysis shows the correlation between analysis shows the correlation between XX and and YY. .

Understanding process, meaning, mechanism of Understanding process, meaning, mechanism of social and human behaviors and phenomenon. social and human behaviors and phenomenon.


Next sessionNext session

More on case studies (by Prof. Paul t’Hart)More on case studies (by Prof. Paul t’Hart) Characteristics of case studiesCharacteristics of case studies When to undertake case study research?When to undertake case study research? Challenges in case study researchChallenges in case study research Sensible description and analysis in a case stSensible description and analysis in a case st

udyudy Bad case study research: traps to avoidBad case study research: traps to avoid Good case study research: tipsGood case study research: tips

5 May 2009 @ Crawford School 1 Comparative Case Studies – 1 Semester 1, 2009 POGO8096/8196:...

Documents

Transcript of 5 May 2009 @ Crawford School 1 Comparative Case Studies – 1 Semester 1, 2009 POGO8096/8196:...