Introduction: Research design

17.871

Spring 2012

1

The Biggest Problem in Research: Establishing Causality Return to the case of voting machine problems

in Florida After the election, we wanted to know: are some

machines “better” than others? For the policy choice, we want to know if

2

The Biggest Problem in Research: Establishing Causality Return to the case of voting machine problems

in Florida After the election, we wanted to know: are some

machines “better” than others? For the policy choice, we want to know if

3

The Biggest Problem in Research: Establishing Causality Return to the case of voting machine problems

in Florida After the election, we wanted to know: are some

machines “better” than others? For the policy choice, we want to know if

4

The Biggest Problem in Research: Establishing Causality Return to the case of voting machine problems

in Florida After the election, we wanted to know: are some

machines “better” than others? For the policy choice, we want to know if

5

The problem

How do we make sure that quality differences observed among machine types are due to machine types per seThis is an issue of causalityWe attend to “internal validity” so that when

we observe differences between groups, we can assure ourselves that this is because of the “treatments” of interest

6

Review of internal and external validity

Internal validity: the two problemsThe two primary threats to internal validity.

1. Nonrandom selection into the treatment group (confounding variables) Comparing

apples with apples or apples with oranges?

Random assignment ensures apple to apple comparisons

Regression, matching, difference-in-differences also attempt to compare apples with apples

2. Reverse causation The chicken and egg problem, which came

first? Is your dependent variable influencing your

treatment (your explanatory variable)? If you can address these problems, you almost

always have an internally valid study Randomly assigned experiments address both

External validity

Is your sample representative of the population? Make sure your study population is relevant to the

general population Address by randomly sampling

Good research is about addressing Internal validity External validity

Clarification

Randomly sampling cases gets you? External validity

Randomly assigning to treatment group? Internal validity

Controlling for variables with regression addresses? Internal validity

What study design addresses both internal and external validity? Field experiments

What is gold standard research design? Field experiment, e.g., Connecticut voting

turnout Why? Addresses

Internal validity Nonrandom selection into the treatment Reverse causation

External validity What aspects of our lives are governed by gold standard

research? In this class, we mostly do observational studies,

But the key to a successful observational research is always keep in mind how one study differs from a field experiment

12

Next class: STATA

Kohler & Kreuter, Data analysis (2nd edition)

Chapter 1 Skip section 1.3.19 (linear regression)

Chapter 3 Only read section 3.1

Chapter 5 Read section 5.1 but skip 5.1.3 and 5.1.4 Read section 5.2

Handout: “How to use the STATA infile and infix commands” (course website)

13

If you want to play around with Stata Visit

http://ist.mit.edu/services/software/athena/numerical for basic info about accessing Stata in Athena

Look on pp. xxi-xxii of Kohler & Kreuter for info about download example data

14