Has Experimental Economics Lived Up To Its Promise?

Alvin E. RothHarvard University

NYU Conference on the Method of Modern Experimental Economics

April 27-28, 2008

This version April 22, 2008

Other topics I could have been asked to speak about

• Has the 21st Century Lived Up To Its Promise?• Have Your Children Lived Up To Their

Promise?– The questions we do ask about our children are

• How have they matured?• What have we learned about them?

• One way we get a snapshot of how our children are thriving is by looking at their latest report card. • Another is by comparing them to their baby

photos. 2

Latest report card: Most recent AER—March 2008

• Philip A. Haile, Ali Hortaçsu and Grigory Kosenok, “On the Empirical Content of Quantal Response Equilibrium”

• Iris Bohnet, Fiona Greig, Benedikt Herrmann and Richard Zeckhauser, “Betrayal Aversion: Evidence from Brazil, China, Oman, Switzerland, Turkey, and the United States”

3

A word about my experimental design• If I hadn’t found appropriate papers in the most recent

AER, I would have looked further.• So I shouldn’t (and you shouldn’t) draw over-broad

conclusions from the fact that I found two suitable papers in the first place I looked (and the only one I report).

• This used to be a bit of a problem in how some experiments were reported (e.g. when each observation of a market was sometimes reported as an experiment…)– Roth, A.E. "

Let's Keep the Con Out of Experimental Econ.: A Methodological Note" Empirical Economics (Special Issue on Experimental Economics), 1994, 19, 279-289.

• But it’s much less of a problem today, a good reporting culture is developing. 4

Betrayal Aversion: Evidence from Brazil, China, Oman, Switzerland, Turkey, and the United States

• Bohnet et al. (2008) builds on two literatures in experimental economics: trust, and “cross cultural” comparisons.

• “To ensure the equivalence of experimental procedures across countries, we followed Roth et al. (1991) on appropriate designs for multinational experiments. Thus, we controlled for currency, language, and experimenter effects to the best of our ability…”

5

6

“Cross cultural” experiments

• Roth, A.E., Prasnikar, V., Okuno-Fujiwara, M., and Zamir, S. "Bargaining and Market Behavior in Jerusalem, Ljubljana, Pittsburgh, and Tokyo: An Experimental Study," AER, 1991.

• The ambitions of Roth et al. 1991 were relatively modest: – to compare bargaining and market games on

different subject pools, – develop experimental methods to do so.

• Since then, more specifically ‘cross cultural’ agendas– Agay et al. (2008), “Ultimatum bargaining behavior

of people affected by schizophrenia,” PSYCHIATRY RESEARCH, 157, 39-46.

– Anthropologists– Public goods and ‘altruistic’ punishment

• Fehr et al, and a variety of different results, e.g. Novak and Fudenberg et al.

7

Bohnet et al.

8

• “Looking at each country relative to the sample, only Oman proved to be an outlier, being more betrayal averse and more risk averse …

• “Our experiments show that people typically sacrifice more expected monetary value to avoid being betrayed than they sacrifice to avoid losing in a lottery offering the same odds and payoffs. …none of the five new nations studied differs significantly in its level of betrayal aversion from the United States.”

• “We consider our approach to be an early, small step toward increasing the “global validity” of laboratory experiments, as we estimate that more than 90 percent of experiments are run with American or Western European subjects.”

9

• Bohnet et al. is a careful crosscultural study that looks at a variable with potential cultural significance.

• Overall, for crosscultural studies, this report card reads “Showing satisfactory progress”

• That’s better than some of the interim report cards; not all the crosscultural studies have done as well.– “needs more control,” – “ jumps to (too broad) conclusions”

• This last is something we’ve seen on many report cards for experimental economics—it bears watching

10

Some consequential subject pool differences?

• Gender• E.g.

– Gneezy, Uri, Muriel Niederle, Aldo Rustichini, “Performance in Competitive Environments: Gender Differences”, Quarterly Journal of Economics, CXVIII, August 2003, 1049 – 1074.

– Niederle, Muriel, and Lise Vesterlund, “Do Women Shy away from Competition? Do Men Compete too Much?,” Quarterly Journal of Economics, August 2007, Vol. 122, No. 3: 1067-1101.

11

• Let’s go back to the most recent report card and look at the next item…

12

Parameters• Haile, Hortaçsu and Kosenok, “On the

Empirical Content of Quantal Response Equilibrium,” AER, March 2008

• They direct our attention to– “The following quotation, from …(2004)…:– ‘Quantal response equilibrium… should

replace Nash as the static benchmark to which other models are routinely compared’.”

13

They conclude (with qualifications I omit from this slide)

• “…evaluating the fit of a QRE in a single game is uninformative without significant a priori restrictions on the distributions of payoff perturbations. This should not be mistaken for a critique of the QRE notion itself. Rather, our aim has been to clarify some limitations of examining behavior one game at a time and to develop approaches for more informative evaluation of QRE.”

• Currently, evidence regarding such comparative statics predictions of QRE is limited, and we hope that attention to this topic here and in Goeree, Holt, and Palfrey (2005) leads researchers to explore richer sets of testable restrictions in order to better evaluate QRE as a tool for understanding and predicting strategic behavior.

14

Report card?

• Overall, for using multi-parameter models to describe experimental data ex post, this report card reads – “Needs improvement”.

• There are some signs of improvement.

15

Prediction

• Ido Erev will speak later about The Technion Prediction Tournament, which is designed to investigate what kinds of models predict behavior well in various kinds of individual choice tasks.

16

Individual choice—comparing current and baby photos

• Prospect theory– Encoded regularities concerning small

probabilities, gains, losses…

• Experience versus description—Erev et al.

• Some conclusions of prospect theory may be more closely related to the way the experiments were performed than was earlier appreciated.

• E.g. small probabilities that are overestimated when described are underestimated when experienced.

17

18

Our youngest children have the most promise, and we can try to learn how to be better parents from experience of older sibs

• Neuroeconomics• Handbook of Experimental Economics, John

Kagel and Alvin E. Roth, eds. Princeton University Press (v1, 1995), v2 (in preparation)– Chapter 8: Neuroeconomics (David Laibson,

Harvard, Colin Camerer (Cal Tech), Jon Cohen (Princeton), Ernst Fehr (Zurich), Paul Glimcher (NYU), George Loewenstein (Carnegie Mellon), Read Montague (Baylor))

Understanding brains offers many opportunities

• But we will have to exercise caution about our tendency for exuberant over-generalization.

19

20

Science has often followed instrumentation

• Telescopes• Microscopes• Psychological/physiological instrumentation

– Tachistiscopes (reaction time)– Eye movement measurement– EEG (evoked brain potential; “brainwaves”)– PET scanners– fMRI

• In each case, new instruments have enabled new kinds of observations and experiments.

• But the pioneers know that they run an extra risk in interpretation of the data from new instruments– Canals on Mars?– Anderson, J. R., Bothell, D., & Douglass, S. (2004) Eye

movements do not reflect retrieval. Psychological Science. 15, 225-231

Eye Movements

• Anderson et al. (2004) Eye Movements Do Not Reflect Retrieval Processes. Limits of the Eye-Mind Hypothesis, Psychological Science 15 (4) , 225–231

• “The eye-mind hypothesis (Just & Carpenter, 1984) states that there is a strong correlation between where one is looking and what one is thinking about.”

21

Eye Movements• Anderson et al. look at a task that requires

you to encode two facts (a person and a location) and then answer a question about them.

• They conclude that, once you have looked at the two, the “retrieval” process is independent of the “gaze” process.– “…the mind has a path to travel after the

encoding of the information, and eye movements do not indicate what is happening on that path.”

22

23

fMRI of a car

3 Hummers, 2 BMWs and 4 minis mapped into normalized Talairach coordinates of a Prius.

24

Oxygen consumptionAverages of differences over different trips

Trips to restaurants

Trips to grandma

all trips

25

Interpretation

• Notice that the hot spots on the car are likely all reliable differences between different kinds of trips.

• But our present understanding of cars allows us to interpret different hotspots differently.

• Experiments focused on the workings of the car would likely concentrate more finely on the engine and mechanical parts, and might select different experimental conditions as trips, e.g. in town versus on the highway, fast versus slow, etc.

• And questions about the use of the car might still be answered more informatively by interviewing the driver and passengers, than by looking at hot spots on passenger seats.

• But these are distinctions that rely on our body of knowledge about cars, which we know more about than brains. – (everyone has a car…)

26

First report card?

Pleasure centers of the brain

27

Market design• Early studies—experiments were expected

to bear a lot of the weight• More recently—experiments serve a more

modest but effective role.– Experiments for hypothesis testing– Experiments for testbeds– Experiments—or experimental demos-- as

demonstrations• E.g. winner’s curse jar of coins, demand

withholding in multi unit auctions

28

Market Design:

The market design “movement,” as I understand it, seeks to:

• Understand design features of existing markets– Experiments help to isolate and vary particular features

• Test new design elements not yet found anywhere– Experiments give us the only empirical data

• Develop and help implement practical market designs, for new markets, and to correct market failures in existing markets.– Experiments play multiple roles

Designing a new market for Gastroenterologists: Experiments complement other kinds of work

Niederle, Muriel and Alvin E. Roth, “Unraveling reduces mobility in a labor market: Gastroenterology with and without a centralized match,” Journal of Political Economy, vol. 111, no. 6, December 2003, 1342-1352.

Niederle, Muriel and Alvin E. Roth, “Relationship Between Wages and Presence of a Match in Medical Fellowships,” JAMA. Journal of the American Medical Association, vol. 290, No. 9, September 3, 2003, 1153-1154.

Niederle, Muriel and Alvin E. Roth, “The Gastroenterology Fellowship Match: How it failed, and why it could succeed once again,” Gastroenterology, 127, 2 August 2004, 658-666.

C. Nicholas McKinney, Muriel Niederle, and Alvin E. Roth, “The collapse of a medical labor clearinghouse (and why such failures are rare),” American Economic Review, 95, 3, June, 2005,878-889.

Niederle, Muriel and Alvin E. Roth, “The Gastroenterology Fellowship Market: Should there be a Match?,” American Economic Review, Papers and Proceedings, 95,2, May, 2005, 372-375.

Niederle, Muriel, and Alvin E. Roth, “Re-starting the Gastroenterology Match,” letter, American Journal of Gastroenterology, vol. 100, no. 5, May, 2005, 1202-1203.

Niederle, Muriel, Deborah D. Proctor, and Alvin E. Roth, “What will be needed for the new GI fellowship match to succeed?” Gastroenterology, January, 2006, 130, 218-224.

Niederle, Muriel and Alvin E. Roth, “Making Markets Thick: Designing Rules for Offers and Acceptances,” working paper.

29

30

Experiments have multiple roles to play

• They allow us to investigate questions that the field data cannot answer– E.g. why did the GI match fail in ’96?

• They allow us to investigate hypotheses suggested by the field data– E.g. does the ability to renege on early acceptances

reduce exploding offers and unraveling?• They play a role in the considerable amounts of

discovery, demonstration, and persuasion that are necessary to coordinate market participants to move from one equilibrium to a better one.

Why experiments help communicate

• Comparison between one complicated market and another is often viewed with suspicion– e.g. – The market for lawyers is very different from the

market for docs – The market for American docs is very different from

the markets for British docs– The market for gastro fellows is very different from the

market for Ph.D. students• Experiments are also very different from complex

markets.– But they are different in simple, transparent ways.

31

32

Critiques

• Critiques of experiments, and critiques of (overbroad) conclusions and models…

33

Experimental econ has matured

• Not so long ago, there wasn’t any criticism of experimental economics, there were just statements to the effect that “economics is not an experimental science.” (If you google that phrase, you can still find some examples.)

• We can learn something by looking at the some critiques, parries, defenses

• Vigorous critiques show we have come of age, in the spirit of Ben Gurion (speaking of Israel): "We will know we have become a normal country when Jewish thieves and Jewish prostitutes conduct their business in Hebrew."

Vital signs

• Not only is the latest report card good• Not only is there clear evidence of long

term intellectual growth• But experimental economics also passes

its latest physical exam:– Experiments are regularly published in most

major journals– Experimenters and labs are found in a steadily

increasing number of Economics departments.

34

Overall report card• “Starting to play well with others”

– Experiments are becoming better integrated with other kinds of economic research

• Two-way feedback with theory• Experiments complement other kinds of empirical work.• The lab has unique advantages, as does field data, and

theoretical models.

• “Needs improvement”– Should avoid tendency to overgeneralize, and to

go it alone.

• “Full of promise”– Experiments are essential when control is needed– Continue to cultivate careful design 35