EXPERIMENTS IN POLITICAL SCIENCE1

S. Erdem AytaçKoç University

March 12, 2015

Much of this material is from lectures by Kenneth Scheve and Thad Dunning, and from Gerber and Green (2012), Angrist and Pischke (2009), Mutz (2011), and several journal articles.

Growth of experiments

Experiments• Experiment: an investigation where the researcher

intervenes in the data-generating-process by purposely manipulating elements of the environment (treatment)

• System under study (individuals/material investigated, nature of manipulations, measurement procedures) is under the control of the investigator

• Observational study: some of these features, and in particular the allocation of individuals to treatment groups, are outside the investigator’s control (Cox and Reid 2000:1)

Randomized (controlled) experiments• A sample of N individuals/units is selected from the population

• Note that this sample may not be random and may be selected according to observables

• This sample is then divided randomly into two groups: the Treatment group and the Control group

• The Treatment group is then treated by stimulus X while the Control group is not. Then the outcome Y is observed and compared for both Treatment and Control groups

• The effect of stimulus X is measured in general by the difference in empirical means of Y between Treatment and Control groups

An example• Social Pressure and Voter Turnout: Evidence from a Large-

Scale Field Experiment (Gerber et al. 2008)

• Why do large numbers of people vote despite the fact that “the casting of a single vote is of no significance where there is a multitude of electors”?

• Citizen duty and social pressure• Voting is widely regarded as a citizen duty• Do people worry that others will think less of them if they fail to

participate in elections?

• Priming voters to think about civic duty while at the same time applying different amounts of social pressure

Research design• Obtain the official state voter list• ~ 80K households (out of ~180K) were sent one of four

mailings encouraging them to vote ahead of the 2006 primary election in Michigan

We’ve got questions

Some questions in applied research in political science are primarily descriptive

• How did inequality change during the twentieth century in advanced industrial democracies?

• Do civil wars last longer in countries with rough terrains?• Are elections for European Parliament more competitive

in the last decade than previously?

We ask questions to make descriptive inference – learn about some unobserved phenomenon on the basis of some set of observed facts

We’ve got questions

But often what we want to know is the answer to a counterfactual question

• Would a particular individual voted if she had been contacted by a campaign worker in person?

• Would a particular individual vote for party X if he had more education?

• Do regional trade agreements increase trade?

We ask questions of the form what would have happened to this person’s/city’s/state’s behavior/outcome if she/it had been subjected to some stimulus, T.

Counterfactual questions

Our questions are often cause-and-effect questions

• Does X cause Y?• If X causes Y, how large is the effect?

This is actually pretty hard to do

An example and some notation will show why it is hard

Potential outcomes modelCounterfactual model of causality (potential outcomes model or Neyman-Rubin model)

• Does college education increase people’s income?

• Di = {0,1} binary random variable indicating college education of individual i

• Yi outcome of interest, a measure of income

• Let us call Y1i the income of an individual if she has college education, irrespective of whether she actually had, and Y0i the income of the same individual if she does not have college education.

• For any individual, there are two potential income variables:

Potential outcome = Y1i if Di = 1

Y0i if Di = 0

The problem of causal inference• We want to know Y1i − Y0i , the causal effect of having college

education for individual i

• Problem: We will never have an individual both with and without college education at the same time

• The fundamental problem of causal inference (Holland 1996) • One can never calculate unit-level causal effects

Observed outcome Yi = Y1i if Di = 1

Y0i if Di = 0

= Y0i + (Y1i - Y0i) Di

What can we do?

We will never know the effect of having college education for a particular individual but we may hope to learn the average effect that it will have on individuals:

E [ Y1i − Y0i ]

E [ Y1i − Y0i ] is the average treatment effect or average causal effect. This equation is defined with reference to the population of interest.

The most common subject of investigation in the social sciences

Naïve estimation

So imagine we have a random sample of the population. Some individuals have college education and others do not.

Look at the difference between average income of individuals with college education and average income of individuals without college education

Observed difference in average income

E [ Yi | Di = 1 ] − E [ Yi | Di = 0 ]

Is E [ Yi | Di = 1 ] − E [ Yi | Di = 0 ] equal to E [ Y1i − Y0i ] ?

Observed outcome Yi = Y1i if Di = 1

Y0i if Di = 0

E [ Y1i | Di = 1 ] − E [ Y0i | Di = 0 ]

Add and subtract E [ Y0i | Di = 1 ] and rearrange

E [ Y1i | Di = 1 ] − E [ Y0i | Di = 1 ] + E [ Y0i | Di = 1 ] − E [ Y0i | Di = 0 ]

Average treatment effect on the treated Selection bias

Selection bias

E [ Y0i | Di = 1 ] ≠ E [ Y0i | Di = 0 ]

• Individuals who “select” themselves to Di = 1 (college education) are likely to be different than others in ways that could affect Y0i

• Better motivated• Wealthier• Educated parents with connections

• Treatment status is not independent of potential outcomes

Random assignment solves selection problem

• Random assignment to treatment and control makes Di

independent of potential outcomes

E [ Yi | Di = 1 ] − E [ Yi | Di = 0 ]

= E [ Y1i | Di = 1 ] − E [ Y0i | Di = 0 ]

= E [ Y1i | Di = 1 ] − E [ Y0i | Di = 1 ]

= E [ Y1i − Y0i | Di = 1 ]

= E [ Y1i − Y0i ]

This was what we sought to learn in the first place!

A (much) more concise approach

Analyses of observational data• Model-based inference – statistical adjustments for potential

confounders• Conditional independence of treatment assignment and potential

outcomes

• Difficult to achieve• Relevant confounding variables must be identified & measured• No one can be sure what the set of confounding factors comprises

Without an experiment, natural experiment, a discontinuity, or

some other strong design, no amount of econometric or

statistical modeling can make the move from correlation to

causation persuasive (Sekhon 2009, 487)

Threats to internal validity• Noncompliance

• Subjects receive a treatment other than the one which they were assigned

• Attrition• Outcome data are missing & systematically related to potential

outcomes

• Interference between units• Observations are influenced by experimental conditions to which

other observations are assigned

Experiments by type – APSR, AJPS, JOP

Online Experiments• Experiments administered over the internet

• Recruitment• Online panels, Online labor markets (e.g., Amazon.com’s

Mechanical Turk), Social networks (e.g., Facebook)• Large and more diverse samples (including representative

samples) than usual convenience samples

Advantages of online experiments• Complex experimental designs are possible

• Multiple variations of multiple factors• May not be practical in traditional paper-format• Computer Assisted Telephone Interview (CATI) and Lab

• CATI – difficult for purposes for explaining the rules• Labs – difficult to reach a diverse sample

Bullock et al. (2013)• Partisan Bias in Factual Beliefs about Politics

• Large differences between Democrats and Republicans in stated attitudes about factual matters

• Real or artificial?• Differences in true beliefs OR “expressive value of offering survey

responses that portray one’s party in a favorable light”

• Test: ask factual questions, pay for correct answers

• YouGov/Polimetrix

Advantages of online experiments• Facilitates complex experimental designs

• Graphics, photos, video, audio stimuli possible

Bailenson, Iyengar et al. (2009)• Facial similarity between voters and candidates causes

influence

• Voters are attracted to candidates who most resemble them on ideology, issue positions, party affiliation

• What about physical resemblance? Are voters attracted to candidates who look like them?

• Facial similarity

Advantages of online experiments• Images might be less obtrusive treatments

• Stimuli: race, gender

• Text-based treatment: why mention race/gender? – intent might be recognized

• Images convey information by themselves

• Games• Economic games• Social games

• Instructions are more easily conveyed, large number of “players”

• (When recruited through an online panel) Payments are more credible, straightforward

• People are accustomed to interactions/communicating with others over the internet

Advantages of online experiments

Advantages of online experiments• Lower levels of socially desirable responding than in

surveys delivered over the phone (Chang and Krosnick 2009)

• Quickly unfolding opportunities for experiments – infrastructure ready

• Collecting data inconspicuously • Time spent answering (e.g., in Qualtrics)• IP addresses – location, housing density, crime rates

Recruiting subjects online• Facebook

• % National population: 45 Argentina, 42 Turkey, 48 UK, 37 France, 27 Germany (Samuels and Zucco 2012)

• Put an ad, conduct the experiment on Qualtrics • Targeting demographics possible (Age group, location, gender,

interests)

• Pay per click to Facebook• Completion rate: 1 in 7 clicks in Brazil, 1 in 40 in Turkey• ~.10$ / click -> ~ 4$/completed survey

Amazon.com’s Mechanical Turk• Web-based platform for recruiting and paying subjects to

perform tasks• Low-cost, easy-to-field platform for survey/experiments

• More than 500,000 individuals from 190 countries (2014)• US, India dominated, less than a quarter in other countries

• Workers are diverse but not representative• Younger, over-educated, under-employed, less religious, more

liberal (Berinsky et al. 2012)

• Results from published studies can be replicated (Berinsky et al. 2012)