Data Analysis

Econ 176, Fall 2014

Populations

When we run an experiment, we are always measuring an outcome, x.

We say that an outcome belongs to some population of outcomes.

• Population of the behavior of undergrad students participating in a particular economic system with a particular set of parameters.

If we changed something about the experiment, we would be looking at a different population.

Usually assume our population is representative in some way.

• Of any human being who could be in the experiment.

• Of outcomes in the economic systems we model with our experiment (external validity).

Populations

Implicit in the idea of a population of observations, is the expectations some variation

• In the population of “students playing the battle of the sexes game” I would expect some people to act differently than others.

We usually want to know what the average outcome is for a population.

Average behavior.

• On average how much do people contribute to public goods?

• On average how often do pairs play the payoff dominant equilibrium?

• On average how often does behavior converge to a competitive equilibrium?• On average how rational are people’s use of imperfect information?

Sampling Error

We are usually blocked from directly observing the actual average of the population (called the population mean)

We usually only get a small sample of the population.

• We don’t get to see everyone’s behavior -- just the few people who participate in our experiment.

As a result, the average behavior we observe could be (by pure chance) different from the actual average behavior overall for the population.

We call this (very real) possibility, sampling error.

• The reason we look at many observations.

Sampling Error

As we get larger and larger samples from a population, we can be more and more sure our average behavior is like the population’s average behavior.

In the limit, sampling error goes away, and we have perfectly reliable results

Statistics says that we can be more confident in our results, the more independent observations we have in our sample.

Independent Trials

What counts as an independent trial?

Technically, an observation (or set of observations) in our sample that doesn’t depend on:

• Other observations from our sample.• Causes that are also causing other observations.

Why not take 1000 observations on one person?

A single subject?

A single group of subjects playing at once?

Informal and Formal Data AnalysisThere are two stages of data analysis: Informal and Formal.

Informal:

Look at summary statistics.

Look at summary graphs.

Look for obvious patterns, tendencies, comparisons.

Formal:

Do statistical analysis.

“Prove” obvious patterns you see.

Look for non-obvious patterns.

Tools

Microsoft Excel (part of Office) has a lot of basic tools.

Analysis Took Pack

Make graphs.

Do statistical tests.

More advanced statistical packages.

Stata

SAS

EViews

MeasurementOften economic theory predicts behavior should look like something specific.

• A first price sealed bid auction bid should be:

€

b =n −1

n

⎛

⎝ ⎜

⎞

⎠ ⎟v

• People should contribute 0 in public goods situations.

• Most of the time, the “payoff dominant equilibrium will be chosen.”

• People shouldn’t exhibit trust in non-repeated interactions.

We often want to look at the data and discover whether behavior in the population matches well with these predictions.

Often we want to measure things that theory tells us nothing much about.

A Bargaining MarketImagine we have six buyers and six sellers with values and costs and no knowledge of the values and costs of others.

The institution for trade: people walk around and negotiate.

Each period they trade until nobody wants to trade anymore.

An experimental question: How efficient was trade?

We run 3 sessions.

Each session consists of 10 periods of trade.

Looking at Averages

Bargaining Markets

59

59.5

60

60.5

61

61.5

62

62.5

63

63.5

1 2 3

Session

Average Efficiency %

We are interested in what tends to happen in the population.

Look at the average efficiency over all rounds for each session.

It looks like something strange was happening in session 1 -- a much larger efficiency?

Looking at Averages

No…the graph simply wasn’t scaled very well.

It looks like average behavior is relatively consistent across sessions.

So, based on this sample, what do we estimate to be in the population?

Bargaining Markets

0

10

20

30

40

50

60

70

80

90

100

1 2 3

Session

Average Efficiency %

Looking at Averages

Based on the data so far, it looks like average efficiency is just over 60%.

We add an error bar -- the standard deviation divided by the square root of observations.

Give us a sense of how much behavior varies across our observations.

Bargaining Markets

0

10

20

30

40

50

60

70

80

90

100

1

Average Efficiency %

Looking at Distributions

Generally we want to get a feel for exactly how much behavior varies.

Tells us how precise our estimates are.

Can use a histogram.

Shows frequency of observations in different ranges.

Histogram of Efficiencies

0

5

10

15

20

25

0 10 20 30 40 50 60 70 80 90 100

Efficiency

Frequency

How dispersed is the data?

Looking at Trends

Often we want to look at how behavior changes over time.

Efficiency by Period

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Period

Efficiency

Session 1

Session 2

Session 3

Take same data from histogram and divide it up by session and period.

Look for:

• Experience effects

• Learning

Looking at Trends

We can look at the average over sessions for each period.

Gives us an estimate in each period rather than overall.

Average Efficiency by Period

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Period

Efficiency

SessionAverage

So it looks like there is some sort of experience effect at work here.

ComparisonWhenever we design a treatment (vary a factor) we do so to compare measurements on two or more populations.

• Every treatment has its own population of possible behavior.

What sort of comparisons do we make?

• Efficiency in double auctions vs. posted offer markets.

• Coordination when battle of the sexes outcomes are close to equal vs. really unequal.

• Effects of chat-room on behavior in game theory games.

• Bidding behavior in first price versus English auctions.

Comparing InstitutionsWhat happens to efficiency when we change the institution to a double auction?

Imagine, again, we have six buyers and six sellers with values and costs and no knowledge of the values and costs of others.

Each period they trade in a double auction

An experimental question: Does the change in the institution alter efficiency?

We run 3 more sessions, under a new institution, with new subjects. Each session, again, consists of 10 periods of trade.

Now we are comparing behavior in treatments.

Comparing Averages

First we can look at the average efficiency in each session, for both treatments.

Bargaining vs. Market

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6

Session

Average Efficiency %

Compare the natural variation within a treatment seems to be similar to variation between treatments.

Does the treatment have too big of an effect to be chalked up to natural variation?

Comparing Averages

We can also look at the overall average in each treatment.

Bargaining vs. Markets

0

10

20

30

40

50

60

70

80

90

100

1

Treatment

Average Efficiency %

Do these averages look different?

The error bars (showing standard deviation) give us some sense of the effect of the treatment relative to normal variation in observed behavior.

Comparing Distributions

Another way to compare the overall variation to the variation due to treatment is to look at side by side histograms.

If they “overlap” too much, we might suspect the behavior in the two treatments come from the same distribution.

That is, the treatment doesn’t have an effect.

Histogram of Efficiencies

0

5

10

15

20

25

0 10 20 30 40 50 60 70 80 90 100

Efficiency

Frequency

We don’t have that “problem” here!

Comparing Trends

We can also compare the trends across periods.

Efficiency by Period

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Period

Efficiency

Session 1

Session 2

Session 3

Session 4

Session 5

Session 6

Do behaviors converge?

Does learning happen in both treatments?

Does one institution require more experience to use well?

Comparing Trends

We can compare the averages over sessions for each period and treatment.

Average Efficiency by Period

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Period

Efficiency BargainingAverageMarketAverage

Don’t converge.

Experience seems to matter for the Bargaining treatment but not for the Market treatment.