Designing Economic Experiments

8/18/2019 Designing Economic Experiments

1/20

219Revista Brasileira de Inovação, Campinas (SP), 11, n. esp., p. 219-238, julho 2012

Designing economic experiments for

evaluation purposes

Marc WillingerFaculté d’Economie, LAMETA, Université de Montpellier 1

Recebido: 28/04/2011 Aprovado: 22/03/2012

A BSTRACT

The growing popularity of experimentation in economics has widened the scope for economic

experiments. In this paper we question the relevance of experimental methods for economic

impact assessment. The major issue of impact evaluation is answering a counterfactual ques-

tion. We show that the economists’ experimental toolbox can provide the appropriate method

to give the right answer, especially by relying on randomized field experiments (RFEs). We

contrast RFEs to other types of experiments, and discuss the limitations of experiments for

evaluation purposes, by presenting 3 case studies that relied on economic experiments at theindividual, local and national levels.

K EYWORDS | Experimental economics; Randomized field experiments; Economic

methodology.

JEL-Codes | C9; C93; H0.

* I would like to thank two anonymous referees for useful comments that helped to improve the paper.


2/20

220

Marc Willinger

Revista Brasileira de Inovação, Campinas (SP), 11, n. esp., p. 219-238, julho 2012

Desenho de experimentos econômicos para fins de avaliação

R ESUMO

A crescente popularidade da experimentação na economia ampliou o espaço para experimentos

econômicos. Neste trabalho questionamos a relevância dos métodos experimentais para

a avaliação de impactos econômicos. A principal finalidade da avaliação de impactos é

responder a uma pergunta contrafactual. Mostramos que a caixa de ferramentas experimental

do economista pode fornecer o método apropriado para dar a resposta certa, sobretudo

a partir do uso de experimentos de campo randomizados (randomized field experiments,

RFE). Contrastamos os RFEs com outros tipos de experimento e discutimos as limitações

dos experimentos para fins de avaliação, apresentando três estudos de caso que fizeram uso

de experimentos econômicos nos níveis individual, local e nacional.

P ALAVRAS-CHAVE | Economia experimental; Experimentos randomizados; Metodologia

econômica.

CÓDIGOS JEL | C9; C93; H0.

Introduction

Economists increasingly rely on experiments to investigate various aspects of their

discipline. While early experiments were tailored for the purpose of theory testing,more recent experiments were designed for evaluating new institutions and markets,testing alternative policy instruments and exploring new areas.

In this paper we question the relevance of experimental methods for economicevaluation. Evaluation covers a large spectrum of activities, including policies, ins-titutions, professionals (researchers, doctors etc.), and impact assessment of socialprograms or science and technology programs, among others. Over the last decadesthe scope of evaluation has been broadened widely and has often been included asa requirement for various programs sponsored by public and private institutions.


3/20


4/20


5/20


6/20

224

Marc Willinger


and settings). By choosing a method of evaluation the researcher is confrontedwith a tradeoff between internal and external validity (ROE; JUST, 2009), which

is illustrated in Figure 1.Laboratory experiments have strong internal validity but there is a cost in terms

of external validity. Control is high and causality can often be clearly established,due to the strong intervention of the researcher. Furthermore, lab experiments arereplicable. But many lab experiments tend to abstract away from the context in orderto reach such control. Very often they involve student subjects whose characteristicsmay systematically differ from those of interest in the field.

Gathering naturally occurring data provides strong external validity, but hasstrong limitations: replicability is often not feasible, control and treatment groupsmay involve systematic differences due to lack of control, causality cannot be clearlyestablished and may be spurious because uncontrolled variables affect the controland/or the treatment group.

FIGURE 1

Tradeoffs across research methodologies according to Roe and Just (2009)

Relative Internal

Validity

Relative Exter-

nal Validity

Topic and

Subject Limits

Replicable?

LabExperiments

High Low Long durationtopics, largerstakes, losses

High

FieldExperiments

Medium toHigh

Medium toHigh

Limited by re-searcherconnections

Low to medium

NaturalExperiments

Medium toHigh

High Limited byoccurrencesof nature andpolicy

Low

Field/marketData

Low High Limited by pri-vacy, recall andtrade secrets

Low to medium

Field experiments offer an attractive compromise. Most of the context is notunder the control of the researcher compared to lab experiments, but the researchercan exogenously manipulate some of the dimensions and observe their impact.Consequently, the data generated by a field experiment reflects both external validityand internal validity. Field experiments therefore represent a bridge between the


7/20


8/20


9/20


10/20

228

Marc Willinger


torically, large-scale social experiments were first developed in the UK to evaluateelectricity pricing schemes (one study started in 1966 and lasted 6 years). In the

US, large-scale social experiments were implemented in the late sixties, followingthe academic and political debate on the US welfare system after the publicationof the Coleman report. Early experiments were targeted towards negative incometaxation, an idea that became popular after Milton Friedman published his bookCapitalism and Freedom (1962). A large-scale program involving 1300 householdswas set up for a period of three years. Individuals assigned to the program wereguaranteed a minimum income and any dollar earned in employment reducedtheir subsidy by 50% of that dollar earned. A broad debate occurred about thework incentives provided by such a program, both before and after the program,reaching a consensus about the negative effect on labor supply compared to thepreexisting welfare system. After this initial experiment in the US more than 235social experiments were conducted.

4. How can field experiments be used for economic evaluation?

Three examples

This section illustrates the usefulness of economic experiments for evaluation pur-

poses by reviewing three examples: the allocation of irrigation quotas at the regionallevel in the US, incentives to increase crop yield at the state level in Africa, andincentives to increase productivity at the firm level in UK. Each study illustratesthe applicability of experimental evaluation at some level: the individual, state ornational level.

4.1 Allocating irrigation quotas

The efficient allocation of water quotas is becoming increasingly imperious in acontext of global warming, especially in arid and semi-arid regions, but more gene-rally in any region which relies heavily on irrigation. In many places, market toolsare used in place of grandfathering allocation of water rights or arbitrary sharingrules (Australia, US). In 2000, the US state of Georgia passed a law requiring theuse of an auction-like mechanism to allocate irrigation rights in periods of drought(the Flint River Drought Protection Act). Economists generally consider auctionsefficient tools for allocation purposes. However, laboratory experiments showedthat depending on the type of auction, misallocations may arise. In the case of


11/20


12/20


13/20


14/20


15/20


16/20


17/20


18/20

236

Marc Willinger


projects that have large spillover effects. In the latter case, experimentation is nolonger the relevant tool for evaluating alternatives, and probably the debate about

which project to implement has to be carried to the political arena.

References

BANDIERA, O.; BARANKAY, I.; RASUL, I. Incentives for managers and inequality amongworkers: evidence from a firm-level experiment. Econometrica , v. 77, n. 4, p. 1047-1094,

July 2009.

_________. Social preferences and the response to incentives: evidence from personnel data.Quarterly Journal of Economics , v. 120, n. 3, p. 917-962, 2005.

CARPENTER, J.; HARRISON, G.; LIST, J. Field experiments in economics: an intro-duction. In: CARPENTER, J.; HARRISON, G. W.; LIST, J. A. (Eds.). Field experiments

in economics . Greenwich, CT: JAI Press, Research in Experimental Economics, v. 10, 2004.

CUMMINGS, R.; HOLT, C.; LAURY, S. Using laboratory experiments for policymaking:

an example from the Georgia irrigation reduction auction. Journal of Policy Analysis and

Management , v. 23, n. 2, p. 341-363, 2004.

DUFLO, E.; KREMER, M.; ROBINSON, J. How high are rates of return to fertilizer?Evidence from field experiments in Kenya. American Economic Review (Papers and Procee-

dings), v. 98, n. 2, p. 482-488, 2008.

_________. Nudging farmers to use fertilizer: theory and experimental evidence from Kenya . Working paper, 2010.

FISHER, R. A. The design of experiments . Edinburgh: Oliver and Boyd, 1935.

FRIEDMAN, M. Capitalism and freedom. University of Chicago Press, 1962.

HARRISON, G. W.; LIST, J. A. Field experiments. Journal of Economic Literature , n. 42,

p. 1009-55, 2004.

HARRISON, G. W. Field experiments and control. In: CARPENTER, J.; HARRISON, G. W.; LIST, J. A. (Eds.). Field experiments in economics . Greenwich, CT: JAI Press, Research

in Experimental Economics, v. 10, 2004.

KARLAN, D. Thoughts on randomized trials for evaluation of development : presentation to

the Cairo evaluation clinic. Yale University, Innovations for Poverty Action, Jameel Poverty Action Lab, 2009.

LEVITT, S.; LIST, J. What do laboratory experiments measuring social preferences tell usabout the real world? Journal of Economic Perspectives , v. 21, n. 2, p. 153-174, 2006.


19/20

237

Designing economic experiments for evaluation purposes


_________. Field experiments in economics: the past, the present, and the future. EuropeanEconomic Review , n. 53, p. 1-18, 2009.

LIST, J. Field experiments: A bridge between lab and naturally occurring data. The B.E. Journal of Economic Analysis & Policy , v. 6, n. 2 - Advances, article 8, 2006.

O’DONOGHUE, R. M. Doing it now or doing it later. American Economic Review , v. 89,

n. 1, p. 103-124, 1999.

ROE, B.; JUST, D. Internal and external validity in economics research: tradeoffs between

experiments, field experiments, natural experiments and field data. American Journal of

Agricultural Economics , v. 91, n. 5, p. 1266-1271, 2009.

ROTH, A. The handbook of experimental economics . John H. Kagel and Alvin E. Roth,

editors, Princeton University Press, v. 1, 1995.SMITH, V. Economics in the laboratory. The Journal of Economic Perspectives , v. 8, n. 1,p. 113-131, 1994.

THALER, R.; SUNSTEIN, C. Nudge: improving decisions about health, wealth, and

happiness. New Haven, CT: Yale University Press, 2008.

CORRESPONDENCE ADDRESS:

Marc Willinger – [email protected], Université de Montpellier 1

Avenue Raymond Dugrand, C.S. 79606


20/20

Designing Economic Experiments

Documents

Transcript of Designing Economic Experiments