Crowdsourcing Contests

Ruggiero CavalloMicrosoft Research NYC

CS286r: November 5, 2012

What is crowdsourcing (for today)?

• Principal seeks production of a good; multiple agents produce; principal obtains value commensurate with highest quality good.

• Examples: logo design, web page design, software development, advice.

• Getting popular on the web – 99designs, Taskcn, Topcoder, Innocentive, CrowdCloud, CrowdFlower, ... Amazon Mechanical Turk, Yahoo! Answers

• And stakes are growing: 99designs.com paid community $1.5 million in January 2012.

2

p

Q1 Q2 Q3u = max{Q1,Q2,Q3}

What is crowdsourcing (for today)?

• Principal seeks production of a good; multiple agents produce; principal obtains value commensurate with highest quality good.

• Examples: logo design, web page design, software development, question answering.

‣ Getting popular on the web – 99designs, Taskcn, Topcoder, Innocentive, CrowdCloud, CrowdFlower, ... Amazon Mechanical Turk, Yahoo! Answers

$ And stakes are growing: 99designs.com has paid out over $40,000,000 to community of 180K designers

3

• number of producers can be very large

• traditionally: only one wins and obtains a “prize”

4

Backing up a little...

• This isn’t quite new, or primarily internet-based.

– Defense contracting (competitors build prototypes, competing for large contract).

– X prize (spacecraft, fuel efficient car, tricorder).

– American Idol?

5

Main existing theory• Contest design in economics (just a sampling):

– [Fullerton and McAfee, 1999]

– [Moldovanu and Sela, 2001, 2006]

• More recently, specifically motivated by online crowdsourcing:

– [DiPalantino and Vojnovic, 2009]

– [Chawla, Hartline, and Sivan, 2012]

– [Archak and Sundararajan, 2009]

– [Cavallo (me) and Jain, 2012]

6

Auctioning Entry Into Tournaments[Fullerton and McAfee, 1999]

• Research tournaments, where participants bear fixed cost plus cost of research effort.

• Principal seeks to maximize best submission net of prize paid out.

– Cost of obtaining a given equilibrium quality level is minimized with 2 participants.

– To get the best participants, conduct a preliminary all-pay auction, which implicitly reveals highest-skilled agents.

7

Auctioning Entry Into Tournaments[Fullerton and McAfee, 1999]

• Research tournaments, where participants bear fixed cost plus cost of research effort.

• Principal seeks to maximize best submission net of prize paid out.

– Cost of obtaining a given equilibrium quality level is minimized with 2 participants.

– To get the best participants, conduct a preliminary all-pay auction, which implicitly reveals highest-skilled agents.

8

99designs.com now similarly has “qualifying” and “final” rounds (where principal chooses up to 6 finalists).

Crowdsourcing and All-Pay Auctions[DiPalantino and Vojnovic, 2009]

• Agents (workers) have private skill, drawn from common-knowledge distribution, which determines how costly it is to produce at a given quality level.

• Agents choose among multiple contests to participate in, and choose effort level.

• In each contest, agent with highest quality submission receives a prize.

– Model equilibrium participation rates as a function of prize-value, compare with empirical data from TaskCN.

9

Optimal Crowdsourcing Contests[Chawla, Hartline, and Sivan, 2012]

• Adopt model of [DiPalantino and Vojnovic, 2009] – analogous to all-pay auction, since all agents pay and only highest “bidder” (quality submitter) obtains the “good” (prize).

• Principal-optimal mechanism design, seeking to maximize either sum of qualities or max quality.

– For sum-of-qualities goal: approximation result (3.164-approx).

– For max-quality goal: winner-take-all is optimal “fixed-prize” format; more messy characterization for the general case.

10

• Almost all previous papers consider the principal’s perspective: how to elicit optimal submission (or sum of submission qualities).

• All (i.e., both of) the main previous computer science papers consider deterministic production.

‣ Rest of the lecture: design of an efficient crowdsourcing mechanism with stochastic production [Cavallo and Jain, 2012].

- optimally trade off benefit to principal with costs to agents

11

When does crowdsourcing make sense?• Two key factors:

1. Uncertain quality of production2. Impatience / deadline

• Otherwise better to just order production sequentially.

12

13

p

Q1 Q2 Q3u = max{Q1,Q2,Q3}

Social welfare = u – agent 1’s production cost – agent 2’s production cost – agent 3’s production cost

Efficient Crowdsourcing Contests[CJ, 2012]: The model

• A principal with private value seeks production of a good.

• A set of agents can individually produce goods.

– Production yields uncertain quality.

– Agents can expend variable privately observed effort; more effort leads to higher expected quality.

– Agents have varying private skill; higher skill leads to higher expected quality.

14

Efficient Crowdsourcing Contests[CJ, 2012]: The model

• A principal with private value seeks production of a good.

• A set of agents can individually produce goods.

– Production yields uncertain quality.

– Agents can expend variable privately observed effort; more effort leads to higher expected quality.

– Agents have varying private skill; higher skill leads to higher expected quality.

15

Will mostly focus on “constant skill” case today.

• Principal has value v ($) for a good with maximum quality

• Agent i with skill si chooses effort δi (which costs $δi)

– a good is produced with quality distributed in a way that depends on v, si, and δi

16

• Principal has value v ($) for a good with maximum quality

• Agent i with skill si chooses effort δi (which costs $δi)

– a good is produced with quality distributed in a way that depends on v, si, and δi

17

Example: quality Qi uniformly distributed between 0 and si δi v

0

1/v

2/v

4/v

v/4 v/2 v

prob

abilit

y de

nsity

quality

uniformly distributed quality

b = 0.25b = 0.5b = 1

• Principal has value v ($) for a good with maximum quality

• Agent i with skill si chooses effort δi (which costs $δi)

– a good is produced with quality distributed in a way that depends on v, si, and δi

18

Example: quality Qi uniformly distributed between 0 and siδiv

0

1/v

2/v

3/v

4/v

0 0.1v 0.3v 0.5v 0.7v 0.9v

prob

abilit

y de

nsity

quality

quality distributed truncated normal

b = 0.1b = 0.3b = 0.5b = 0.7b = 0.9

Example: quality Qi distributed normal with mean si δi v

• Principal has value v ($) for a good with maximum quality

• Agent i with skill si chooses effort δi (which costs $δi)

– a good is produced with quality distributed in a way that depends on v, si, and δi

19

Example: quality Qi uniformly distributed between 0 and siδiv

0

1/v

2/v

3/v

4/v

0 0.1v 0.3v 0.5v 0.7v 0.9v

prob

abilit

y de

nsity

quality

quality distributed truncated normal

b = 0.1b = 0.3b = 0.5b = 0.7b = 0.9

Example: quality Qi distributed normal with mean si δi v

Seek to implement efficient effort policy, maximizing principal’s obtained value minus sum of agents’ costs (effort).

We are concerned with the socially optimal choice of e!ortlevel for each agent, where in light of our utility model theappropriate optimization is the maximum quality level ofthe goods produced minus total e!ort expended; i.e., lettingQi(v, si, !i) be a random variable representing the qualitylevel produced by i ! I who has skill level si and expendse!ort !i (with v the principal’s value), we seek to maximize:

[maxi!I

Qi(v, si, !i)]"!

i!I

!i (3)

An e!ort policy is a function of the principal’s value andthe agents’ skill levels. We let !"(v, s) denote an e!cient pol-icy, i.e., a vector of e!ort levels that maximizes Eq. (3) givenvalues of v and s = (s1, . . . , sn); when context is clear wewill write !"i as shorthand for !"i (v, s). Given our quasilinearutility model, a policy !" that maximizes Eq. (3) maximizesthe expected sum of utilities and is Pareto e"cient.

At various points we will consider a restricted settingwhere skill is constant (and publicly known) throughout thepopulation of agents; we call this the constant skill case.

2. EFFICIENT EFFORT POLICIESIn this section we address the problem of computing an

e"cient policy given full knowledge of the principal’s valuev and agent skill levels s, and given that agents will ex-ecute the e!ort policy that is prescribed. We defer toSection 3 the question of how to implement such a pol-icy in the context of a principal and agents that are self-interested and strategic. We will make heavy use of thefollowing lemma, which demonstrates su"cient conditionsunder which extreme-e"ort policies—those that involve onlytotal (1) or null (0) e!ort by each agent—are optimal. In thissection we make the technical assumption that the cumula-tive distribution over quality, evaluated at any particularquality level, is di!erentiable with respect to e!ort !i.

Lemma 1. For arbitrary i ! I with arbitrary skill si, forarbitrary skill and e"ort levels of the other agents and valuev for the principal, fixing an arbitrary e"ort policy for agentsother than i, amongst e"ort levels within arbitrary interval[a, b] # [0, 1] it is either optimal for i to expend e"ort !i = aor optimal for i to expend e"ort !i = b if the following holds:$" ! [0, v], $# ! [a, b),

" $$!i

"# v

!

F vsi,"i(x) dx

$%%%"i=#

% 1 (4)

& " $$!i

"# v

!

F vsi,"i(x) dx

$%%%"i=k

% 1, $k ! [#, b] (5)

Proof. For arbitrary agent i ! I , consider arbitrary "representing the maximum quality that is to be realized bythe production of the other agents—this is a priori unknown,but a result for arbitrary " will demonstrate that regard-less of its realization the result holds. Then the expectedmarginal impact on e"ciency from i exerting e!ort !i equals:

# v

!

fvsi,"i(x)(x" ") dx" !i (6)

= (x" ")F vsi,"i(x)

%%%x=v

x=!"

# v

!

F vsi,"i(x) dx" !i (7)

= v " " "# v

!

F vsi,"i(x) dx" !i (8)

The first step above is integration by parts. To find themaximum with respect to e!ort, we consider the derivativewith respect to !i, i.e.,

$$!i

"v " " "

# v

!

F vsi,"i(x) dx" !i

$(9)

= " $$!i

"# v

!

F vsi,"i(x) dx

$" 1 (10)

If as !i increases this derivative never changes from pos-itive to negative (this is the condition of the Lemma, inEqs. (4) and (5)), then the maximum lies at one of the ex-tremes, !i = a or !i = b, which completes the proof.

Corollary 1. For environments where the quality dis-tribution functions satisfy the relationship of Eqs. (4–5) inLemma 1 over the full range of e"ort levels ([a, b] = [0, 1]),an e!cient e"ort policy consists of full-e"ort participationby a subset of the agents and non-participation by the others.

Note that the lemma holding for the interval [0, 1] is su"-cient but not necessary for the optimal policy to involve onlyextreme-e!ort (i.e., e!ort 0 or 1 by all agents). For instanceif the condition of the lemma (Eqs. (4) and (5)) does nothold for some ", yet $!i the expected quality output for iis less than " " !i, then the optimal policy would involvenon-participation (0 e!ort) by i when other agents achievequality ", and so it may still be the case that an optimalpolicy never involves intermediate e!ort.

For any constant skill environment (say skill equals s foreach agent) where we can establish that an extreme-e!ortpolicy is optimal, fully determining an e"cient policy is easy.We simply need to compute:

m" = argmaxm!{0,...,n}

&m

# v

0

F vs,1(x)

m#1fvs,1(x)x dx"m

'(11)

m" agents will participate with full-e!ort and the other n"m" will not participate (i.e., will apply 0 e!ort).

When skill is not constant, by Eq. (2) having an agent par-ticipate who has less skill than one who does not participatecould never be optimal. So more generally, for any settingwhere extreme-e!ort has been established as e"cient we candetermine a precise optimal policy by iteratively consider-ing each agent in decreasing order of skill, accepting agentsfor (full-e!ort) participation until stopping and accepting nomore in the ordered list.

In the rest of the section we will show that extreme-e!ortpolicies are optimal in important canonical settings, but wefirst observe that this is not universally the case. Imaginethat e!ort !i ! [0, 0.05) yields quality 0 (with certainty), !i ![0.05, 0.3) yields quality 0 with probability 0.8 and quality0.9 with probability 0.2, and !i ! [0.3, 1] yields quality 0.6with probability 0.8 and quality 0.9 with probability 0.2. Anoptimal policy for two agents has one agent expend e!ort 0.3and the other expend e!ort 0.05.

2.1 Uniformly distributed qualityWe now look at specific distributional settings, starting

with one in which quality is uniformly distributed between0 and the product of the principal’s value and the agent’sskill and e!ort. That is, F v

si,"i for each i ! I is the uniformdistribution over [0, !isiv], i.e.,

fvsi,"i(x) =

(1

"isivif x ! [0, !isiv]

0 otherwise(12)

• Quality Qi – dollar value to the principal of good that i produces – is a stochastic function of v, δi, and si.

• Social welfare equals: max{Q1,Q2,Q3} – δ1 – δ2 – δ3

• But since v and si are private, and δi are privately observed, we need to incentivize principal and agents.

p

v

Q1 Q2 Q3

v,δ1,s1 v,δ2,s2 v,δ3,s3

p

v

Q1 Q2 Q3

v,δ1,s1 v,δ2,s2 v,δ3,s3

1. A computational component:– Determine an effort policy that is efficient, i.e., maximizes sum

of utilities (principal and agents).

2. An incentive component:– A payment mechanism that brings execution of such a policy

into equilibrium.

Efficient crowdsourcing involves:

Efficient effort policy

• In many cases, extreme-effort policies are optimal: each agent exerts either 0 effort or maximal effort.

• If extreme-effort policy is efficient, then determining efficient policy reduces to choosing number of participants.

22

Uniformly distributed quality

Theorem. For the constant skill, uniformly distributed quality case, a mechanism that elicits maximum-effort participation by m* agents (and 0-effort participation by others) is efficient, where:

We call this the uniformly distributed quality case. The rangeof possible qualities (and the expected quality) increases lin-early with skill and e!ort. We can use Lemma 1 to show thatan extreme-e!ort policy is optimal here.

Lemma 2. For the uniformly distributed quality case, forarbitrary skill levels, there is an e!cient policy in which eachagent i ! I exerts either no e"ort !i = 0 or full e"ort !i = 1.

Proof. Consider arbitrary agent i ! I and arbitrary" ! [0, siv]. If we can show that e!ort !i = 0 or !1 = 1 for iwould yield optimal expected marginal e"ciency even if weknew that the other agents would achieve quality ", thenthe theorem follows. Note that in the case of uniformly dis-tributed quality no e!ort level less than or equal to !

sivcould

possibly be optimal because there would be 0 probability ofimproving the maximum quality over " (and so if " > sivthen 0 e!ort by i is clearly optimal). We will now prove theresult by using Lemma 1. For arbitrary !i ! [ !

siv, 1]:

! v

!

F v"i(x) dx =

! "isiv

!

x!isiv

dx+

! v

"isiv

1 dx (13)

=x2

2!isiv

"""x="isiv

x=!+ v " !isiv (14)

=!2i s

2i v

2

2!isiv" "2

2!isiv+ v " !isiv (15)

= v " !isiv2

" "2

2!isiv(16)

Then:

" ##!i

#! v

!

F vsi,"i(x) dx

$=

siv2

" "2

2siv!2i, (17)

and this is at least 1 if and only if:

!2i # "2

2siv " (siv)2(18)

If this holds for !i = $ # !siv

it holds for !i = k for allk > $. Therefore the distribution satisfies Lemma 1, whichtells us that the optimum over the range [ !

siv, 1] occurs at

either !siv

or 1. Since no e!ort level on the interval (0, !siv

]could yield positive expected utility and thus be better thane!ort level 0, the global optimum lies at either !i = 0 or!i = 1, which completes the proof.

Then in the constant skill case (with skill normalized to1) we can compute the optimal number of participants asfollows.

Theorem 1. For the constant skill, uniformly distributedquality case, a mechanism that elicits maximum-e"ort par-ticipation by m! arbitrary agents (and 0-e"ort participationby others) is e!cient, where:

m! =

%$%v& " 1 if $

%v&2 + $

%v& > v

$%v& otherwise

(19)

Proof. By Lemma 2 an optimal policy will involve fulle!ort by some number m ! {0, . . . , n} agents and 0 e!ortby the other n"m agents. If m agents participate with full-e!ort the expected e"ciency equals the expected maximumof m draws from U [0, v] minus m, i.e.:

! v

0

1vmx

#xv

$m"1dx"m =

mm+ 1

v "m (20)

Though m is discrete, imagining it as a continuous variableyields a parabola with a single maximum. Taking the deriva-tive with respect to m, we get v

(m+1)2"1, which has a single

positive root at m =%v " 1. But since m can only take in-

teger values, when%v is not an integer we have to consider

both $%v"1& and '

%v"1( (i.e., $

%v&). The expected value

increase of adding a $%v&th agent to a group of $

%v " 1&

equals, considering Eq. (20):&

$%v&

$%v& + 1

v " $%v&'"

&$%v& " 1

$%v&

v " ($%v& " 1)

'(21)

=1

$%v&($

%v&+ 1)

v " 1 (22)

This is at least 0 if and only if $%v&2 + $

%v& ) v, and so

the maximum is as characterized by Eq. (19).

Figure 1 provides a graphic depiction of how the optimalnumber of agents that should participate relates to the prin-cipal’s value, for v ! [0, 100].

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50 60 70 80 90 100

optim

al n

umbe

r of f

ull-e

ffort

parti

cipa

nts

v

m* as a function of v

Figure 1: Optimal number of agents to produce agood (with full e!ort) in the uniformly distributedquality case, as a function of the principal’s value v.

2.2 Normally distributed qualityWhile for more complex distributions beyond the uni-

form case we would have di"culty demonstrating similarresults analytically, we can query whether Lemma 1 holdsexperimentally. We now consider quality that is normallydistributed, over a bounded interval, with mean increasingproportional to e!ort and skill. Specifically, consider thetruncated normal distribution over the interval [0, v], withlocation parameter µ equal to !isiv and scale parameter %equal to v/8; this distribution is illustrated in Figure 2 forvarious e!ort levels.

For arbitrary v, ", and !i (assuming a fixed si = 1)we can computationally approximate

( v!F vsi,"i

(x) dx and ac-cordingly evaluate whether the conditions of Lemma 1 aresatisfied. But a more direct approach is equally tractablehere: we can simply compute the expected marginal e"-ciency, given any ", of e!ort for arbitrarily fine discretiza-tions of the e!ort space !i ! [0, 1]. Then, the next step isto use this approach to check whether extreme-e!ort is op-timal for all " in the range [0, v], as this would imply thatregardless of the quality obtained by agents other than ar-bitrary i ! I , for i extreme-e!ort (0 or 1) is optimal. We

23

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50 60 70 80 90 100

optim

al n

umbe

r of f

ull-e

ffort

parti

cipa

nts

v

m* as a function of v

Uniformly distributed quality

24

0

1

2

3

4

5

0 10 20 30 40 50 60 70 80 90 100

optim

al n

umbe

r of f

ull-e

ffort

parti

cipa

nts

v

m* as a function of v

Normally distributed quality µ=δiv, σ=v/8

25

Never achieved in eq. with winner-take-all prize structure.

Now for the incentives

26

0

1

2

3

4

5

0 10 20 30 40 50 60 70 80 90 100

optim

al n

umbe

r of f

ull-e

ffort

parti

cipa

nts

v

m* as a function of v

This is what we want to achieve. But can we?

Mechanism design

• The study of how to engineer incentives leading to desirable outcomes despite agent selfishness plus private information and/or autonomy.

• A few examples: auctions for allocating scarce resources; taxation to achieve a desired level of consumption; commissions to achieve sales performance.

27

δ1 δ2 δ3

Make payments to get principal to report true value, and agents to exert prescribed amount of effort.

$$ $

$

p

v

δ1 δ2 δ3

Make payments to get principal to report true value, and agents to exert prescribed amount of effort.

$$ $

$

$

p

v

Goal: a mechanism that is...

• Incentive compatible: no one can benefit from deviating from honest participation

• Individually rational: everyone expects non-negative utility from participating honestly

• No-deficit: the mechanism cannot make positive aggregate payments

30

Mechanism for constant skill setting

• The principal reports v.

• Efficient effort levels δ1, ... , δn are computed.

• Each agent i is instructed to expend effort δi, and goods are produced with quality levels Q1, ..., Qn.

31

Mechanism for constant skill setting: payments

– The principal is charged: agents’ aggregate prescribed effort (δ1 + δ2 ... + δn).

– Each agent is paid:

prescribed effort level + (highest quality level produced overall – highest quality level produced by other agents)

– Each agent is charged:

E[highest quality level overall – highest quality level produced by other agents]

32

Theorem. This mechanism is efficient, incentive compatible, individually rational, and no-deficit in expectation for constant skill settings.

33

Mechanism for constant skill setting: incentives

– The principal is charged: agents’ aggregate prescribed effort.

– Each agent is paid:

prescribed effort level + (highest quality level produced overall – highest quality level produced by other agents)

– Each agent is charged:

E[highest quality level overall – highest quality level produced by other agents]

34

has no effect on incentives

Mechanism for constant skill setting: incentives

– The principal is charged: agents’ aggregate prescribed effort.

– Each agent is paid:

prescribed effort level + (highest quality level produced overall – highest quality level produced by other agents)

– Each agent is charged:

E[highest quality level overall – highest quality level produced by other agents]

35

has no effect on incentives

Principal’s utility is:

highest quality level produced– aggregate effort expended

Agent’s utility is (proportional to):

highest quality level produced– aggregate effort expended

Mechanism for constant skill setting: budget

– The principal is charged: agents’ aggregate prescribed effort.

– Each agent is paid:

prescribed effort level + (highest quality level produced overall – highest quality level produced by other agents)

– Each agent is charged:

E[highest quality level overall – highest quality level produced by other agents]

36

Through this point, budget deficit equals quality difference between top two submissions.

Mechanism for constant skill setting: budget

– The principal is charged: agents’ aggregate prescribed effort.

– Each agent is paid:

prescribed effort level + (highest quality level produced overall – highest quality level produced by other agents)

– Each agent is charged:

E[highest quality level overall – highest quality level produced by other agents]

37

Now budget is balanced in expectation.

Example (uniformly distributed quality)

p

v=8

0

1/v

2/v

4/v

v/4 v/2 v

prob

abilit

y de

nsity

quality

uniformly distributed quality

b = 0.25b = 0.5b = 1

We call this the uniformly distributed quality case. The rangeof possible qualities (and the expected quality) increases lin-early with skill and e!ort. We can use Lemma 1 to show thatan extreme-e!ort policy is optimal here.

Lemma 2. For the uniformly distributed quality case, forarbitrary skill levels, there is an e!cient policy in which eachagent i ! I exerts either no e"ort !i = 0 or full e"ort !i = 1.

Proof. Consider arbitrary agent i ! I and arbitrary" ! [0, siv]. If we can show that e!ort !i = 0 or !1 = 1 for iwould yield optimal expected marginal e"ciency even if weknew that the other agents would achieve quality ", thenthe theorem follows. Note that in the case of uniformly dis-tributed quality no e!ort level less than or equal to !

sivcould

possibly be optimal because there would be 0 probability ofimproving the maximum quality over " (and so if " > sivthen 0 e!ort by i is clearly optimal). We will now prove theresult by using Lemma 1. For arbitrary !i ! [ !

siv, 1]:

! v

!

F v"i(x) dx =

! "isiv

!

x!isiv

dx+

! v

"isiv

1 dx (13)

=x2

2!isiv

"""x="isiv

x=!+ v " !isiv (14)

=!2i s

2i v

2

2!isiv" "2

2!isiv+ v " !isiv (15)

= v " !isiv2

" "2

2!isiv(16)

Then:

" ##!i

#! v

!

F vsi,"i(x) dx

$=

siv2

" "2

2siv!2i, (17)

and this is at least 1 if and only if:

!2i # "2

2siv " (siv)2(18)

If this holds for !i = $ # !siv

it holds for !i = k for allk > $. Therefore the distribution satisfies Lemma 1, whichtells us that the optimum over the range [ !

siv, 1] occurs at

either !siv

or 1. Since no e!ort level on the interval (0, !siv

]could yield positive expected utility and thus be better thane!ort level 0, the global optimum lies at either !i = 0 or!i = 1, which completes the proof.

Then in the constant skill case (with skill normalized to1) we can compute the optimal number of participants asfollows.

Theorem 1. For the constant skill, uniformly distributedquality case, a mechanism that elicits maximum-e"ort par-ticipation by m! arbitrary agents (and 0-e"ort participationby others) is e!cient, where:

m! =

%$%v& " 1 if $

%v&2 + $

%v& > v

$%v& otherwise

(19)

Proof. By Lemma 2 an optimal policy will involve fulle!ort by some number m ! {0, . . . , n} agents and 0 e!ortby the other n"m agents. If m agents participate with full-e!ort the expected e"ciency equals the expected maximumof m draws from U [0, v] minus m, i.e.:

! v

0

1vmx

#xv

$m"1dx"m =

mm+ 1

v "m (20)

Though m is discrete, imagining it as a continuous variableyields a parabola with a single maximum. Taking the deriva-tive with respect to m, we get v

(m+1)2"1, which has a single

positive root at m =%v " 1. But since m can only take in-

teger values, when%v is not an integer we have to consider

both $%v"1& and '

%v"1( (i.e., $

%v&). The expected value

increase of adding a $%v&th agent to a group of $

%v " 1&

equals, considering Eq. (20):&

$%v&

$%v& + 1

v " $%v&'"

&$%v& " 1

$%v&

v " ($%v& " 1)

'(21)

=1

$%v&($

%v&+ 1)

v " 1 (22)

This is at least 0 if and only if $%v&2 + $

%v& ) v, and so

the maximum is as characterized by Eq. (19).

Figure 1 provides a graphic depiction of how the optimalnumber of agents that should participate relates to the prin-cipal’s value, for v ! [0, 100].

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50 60 70 80 90 100

optim

al n

umbe

r of f

ull-e

ffort

parti

cipa

nts

v

m* as a function of v

Figure 1: Optimal number of agents to produce agood (with full e!ort) in the uniformly distributedquality case, as a function of the principal’s value v.

2.2 Normally distributed qualityWhile for more complex distributions beyond the uni-

form case we would have di"culty demonstrating similarresults analytically, we can query whether Lemma 1 holdsexperimentally. We now consider quality that is normallydistributed, over a bounded interval, with mean increasingproportional to e!ort and skill. Specifically, consider thetruncated normal distribution over the interval [0, v], withlocation parameter µ equal to !isiv and scale parameter %equal to v/8; this distribution is illustrated in Figure 2 forvarious e!ort levels.

For arbitrary v, ", and !i (assuming a fixed si = 1)we can computationally approximate

( v!F vsi,"i

(x) dx and ac-cordingly evaluate whether the conditions of Lemma 1 aresatisfied. But a more direct approach is equally tractablehere: we can simply compute the expected marginal e"-ciency, given any ", of e!ort for arbitrarily fine discretiza-tions of the e!ort space !i ! [0, 1]. Then, the next step isto use this approach to check whether extreme-e!ort is op-timal for all " in the range [0, v], as this would imply thatregardless of the quality obtained by agents other than ar-bitrary i ! I , for i extreme-e!ort (0 or 1) is optimal. We

Example (uniformly distributed quality)

• Optimal policy has 2 agents exert full effort.

p

Q1=3v=8

Q2=5 Q3=0

δ1 = 1 δ2 = 1 δ3 = 0

Example (uniformly distributed quality)

p

Q1=3v=8

Q2=5 Q3=0

Must pay: δ1 + δ2 + δ3 = 2

δ1 = 1 δ2 = 1 δ3 = 0

• Optimal policy has 2 agents exert full effort.

p

Q1=3

Examplev=8

Q2=5 Q3=0

Must pay: δ1 + δ2 + δ3 = 2

δ1 = 1 δ2 = 1 δ3 = 0

(uniformly distributed quality)

effort +1 +1 +0

qual. diff +(5 – 5) +(5 – 3) +(5 – 5)

E[qual. diff] –(16/3 – 4) –(16/3 – 4) –(16/3 – 16/3)

is paid = –1/3 = 5/3 = 0

utility –4/3 2/3 0

Revenue = 2 + 1/3 – 5/3 = 2/3Each’s utility is non-negative in expectation, but not guaranteed.

utility = 5 – 2 = 3

In some cases, can charge principal entry fee, redistribute to agents to decrease odds of loss.

Private skill setting is more challenging.

42

Private skill

43

Theorem. There exists no mechanism that is efficient, incentive compatible, individually rational, and no-deficit in expectation.

However:

• In some cases weaker individual rationality concept may suffice.

– Require commitment prior to revealing nature of task.

44

Efficient mechanism that is incentive compatible, no-deficit, and satisfies this notion of individual rationality.

p

v=8s1 ∼ U(0,1) s2 ∼ U(0,1) s3 ∼ U(0.5,1)

p

v=8s1 = 0.7 s2 = 0.2 s3 = 0.5

reveal task

public knowledge:

private knowledge:

will participate regardless of v will “sign up” if forced to choose now

may or may not regret having participated

II. Mechanism for private skill setting

– The principal is charged: agents’ aggregate prescribed effort, plus G.

– Each agent is paid: highest quality produced, minus effort prescribed for other agents, minus a balancing term independent of reported skill levels, plus 1/n times G.

Let G equal minimum possible “expected value” for principal, given distribution over skill levels, and effort-to-quality distribution.

46

Theorem. The mechanism is incentive compatible, individually rational for the principal, individually rational for each agent ex ante of skill level realizations, and no-deficit in expectation ex ante of skill realizations.

Private skill mechanism

47

Summary of [C&J, 2012]+ Efficient mechanisms for crowdsourcing, very

generally applicable

+ more efficiency ➞ more attractive marketplace

– Lacks the simplicity of winner-take-all approach

– in fact “contest” is now a misnomer

• Computing efficient policies can be hard (but can tractably handle lots of natural special cases)

• Important question: how big is the social welfare gain is in relevant cases?

48

High-level recap

• Long line of work in economics considering optimal contest design from perspective of principal, with recent contributions from [CHS, 2012] and [AS, 2009].

– Significant assumptions that are questionable in typical web settings (deterministic production?).

• Then, [CJ, 2012] tries to take perspective of marketplace designer seeking to maximize social welfare, attracting principals and agents.

– Incentives analysis is very generally applicable.

– Computing optimal policies in the general case is hard and deserves more attention.

49