CS483/683 Multi-Agent Systems

Lectures 9-10:Distributed Constraint Optimization:

Auction-based solutions

16 February 2009Instructor: Kostas Bekris

Computer Science & Engineering, University of Nevada, Reno

CS483

Contract Nets

•Assume you have a global problem that consists of multiple tasks

- assign the tasks to different agents

•Each agent has different capabilities

•∀ agents i, ∃ ci(T): cost for agent i to execute all tasks in the set T

•Agents start with an initial assignment and negotiate in order to get an optimum assignment

•Negotiation

- Agents repeatedly contract tasks one to another

✓e.g., exchange tasks for money or tasks for other tasks

- Questions

✓How does the bidding process work?

✓Which contracts are successful given the bidding process?

•Requirement for an anytime solution

CS483

Negotiation

One simple approach:

•Each agent bids for a task the cost it would incur to execute it - c(i)

•Tasks are allocated to lowest bidders

•Repeat

It can be shown that there always exists a sequence of contracts that results in the optimal allocation but...

•if each agent contracts out a single task for money then the optimal sequence might contain “money-losing” contracts

•there are however more complex contracts

- contracting a bundle of tasks (“cluster contracts”)

- swap tasks among 2 agents (“swap contracts”)

- simultaneous transfers among many agents (“multi-agent contracts”)

•the more complex contracts allow for optimal sequences without “money-losing” contracts

CS483

Challenges

1. What is the connection between the following 2 contradictory notions:

•Solving a common global problem

•Minimize the total cost to each individual agent

➡We will see that finding an optimal solution is closely related to the individual utilities of agents

2.When do agents make offers? What is the method by which the contracts are decided upon?

➡We will have to specify the following:

1.Bidding rules: Permissible ways of making offers

✓Market clearing rules: Definition of the outcome based on the offers

✓Information dissemination rules: the information made available to the agents throughout the process

➡We will assume an explicit centralized component

‣ an auctioneer

CS483

Challenges

3.Since we have cooperative agents, why does it matter if agents “lose-money”?

•Auctions in general take place so that each agent will maximize a personal payoff

•Game-theoretic notions

➡In the optimization context:

• agents do not deviate from their prescribed bidding protocol for personal gains

• but game-theoretic agents can be used as a guidance

We will study 2 classical optimization problems:

• Task Assignment problems (linear program)

• Scheduling problems (integer linear program)

and how we can solve them with auctions and negotiations

CS483

An Intro to Linear and Integer Programming

Linear Program:

1.Real-valued variables:

- {x1,…,xn} with each xi ∈ R

•Linear objective function given constants w1, …, wn:

- maximize ∑i wixi

1.minimization problems can be solved with the weights all being negative

•A set of linear constraints given constants a1j,…,anj and a constant bj:

- subject to ∑i aijxi ≤ bj ∀ j = 1, …, m

1.xi ≥ 0 ∀ i = 1, ..., n

CS483

Linear Programming

Similar problems:

•minimization problems:

- all the weights should be negative

•greater-than-or-equal constraints

- negate all the constants in constraints

•equality constraints

- satisfy both less-than-or-equal and greater-than-or-equal constraints

•ignore variables

- set corresponding constants to zero

•piecewise-linear constraints (e.g., max function)

- can be constructed by defining additional constraints and fake variables

CS483

Linear Programming

Matrix form

•w: n × 1 vector containing the weights wi

•x: n × 1 vector containing the variables xi

•A: m × n matrix of constants aij

•b: m × 1 vector of constants bj

maximize wTx

subject to Ax ≤ b

x ≥ 0

Dual problem:

minimize bTy

subject to ATy ≥ w

y ≥ 0

CS483

Economic Interpretation

Imagine a production economy:

•set of resources

•set of products

•each product consumes a certain amount of each resource

•each product is sold at a certain price

Interpret xi as the amount of product i produced and ci as the price of product i:

•maximize profit ∑i cixi

Interpret bj as the available amount of resource j and aij as the amount of resource j needed to produce a unit product i:

•satisfy ∑i aijxi ≤bj

Dual variable y: shadow price of resource i

•High shadow price means that a resource has a high impact on the optimal solution

CS483

Linear Programming

Each linear constraint corresponds to:

• a hyper-plane in an n-dimensional space

Feasible solutions:

•the region bounded by all the hyperplanes (a convex polyhedron)

Because the objective function is also linear:

•any local optimum in the feasible region is a global optimum

•at least one optimal solution will exist at a vertex of the polyhedron

Popular solution: Simplex algorithms

•start from a vertex of the polyhedron

•start moving uphill (i.e., improve the objective function) towards other vertices

•until an optimum is found

Exponential complexity but efficient in practice

CS483

Integer Programming

Like linear programs

•only that the variables have to be... integers!

•much harder computational problem (worst case: undecidable)

A useful subclass: 0-1 integer programs

•variables can only take 0 and 1 integer values

Any problem in the NP class is equivalent to an Integer programming problem

Another useful subclass: mixed-integer program

•some variables can be reals, some must be integers

There is no real efficient solution for integer programs

•commonly used: branch-and-bound search (variation of backtracking search)

- difference from backtracking: additional pruning takes place

✓if the linear program relaxation does not allow certain integer values then these values are not considered in the search procedure

CS483

1. Assignment Problem

a.k.a. weighted matching in a bipartite graph

•a set N of n agents

•a set X of n objects

•a set M ⊆ N × X of possible assignment pairs; and

•a function υ: M ⇒ R giving the value of each assignment pair

An assignment is a set of pairs S ⊆ M so that each agent i ∈ N and each object j ∈ X is in at most one pair in S.

A feasible assignment is one in which all agents are assigned an object

A feasible assignment is optimal if it maximizes:

∑(i,j)∈S u(i,j)

CS483

Example

Utilities

ii u(i,xu(i,x11)) u(i,xu(i,x22)) u(i,xu(i,x33))

1 2 4 0

2 1 5 0

3 1 3 2

Optimal assignment: (1,x1), (2,x2), (3,x3)

CS483

Encoding as a Linear Program

Define indicator matrix x:

•xi,j = 1 indicates that the pair (i,j) is selected

•xi,j = 0 otherwise

Express the linear program as follows:

maximize ∑(i,j)∈M u(i,j)xi,j

subject to ∑j|(i,j)∈M xi,j ≤ 1 ∀ i ∈ N ∑i|(i,j)∈M xi,j ≤ 1 ∀ j ∈ X

Why linear program and not an integer program though?

Turns out:

• the LP formulation allows integral solutions

•any optimal fractional solution can be turned into an integral one

CS483

Properties of Linear Programming Formulation

Since any LP can be solved in polynomial time

•the assignment problem can be solved in polynomial time as well

But!

•the complexity is O(n3), which may be too high in some cases

•no obvious parallelization

•not adaptive to changes in the problem specification

So, lets take a different approach... one that has some competitive notions

CS483

Competitive Equilibrium

Imagine each object in X has an associated price:

•price vector p = (p1, …, pn), where pj is the price of object j

Given an assignment S ⊆ M and a price vector

•redefine the utility from an assignment of object j to agent i as:

- u(i,j) = u(i,j) - pj

Then we can define the notion of a competitive equilibrium:

A feasible assignment S and a price vector p are in competitive equilibrium when for every pairing (i,j) ∈ S it is the case that ∀ k,

u(i,j) ≥ u(i,k)

If a feasible assignment S and a price vector p satisfy the competitive equilibrium condition then S is an optimal

assignment.

For any optimal solution S, ∃ price vector p s.t. p and S satisfy the competitive equilibrium condition.

CS483

Price Vector example

Price Vector: (2,4,1)

ii u(i,xu(i,x11)) u(i,xu(i,x22)) u(i,xu(i,x33))

1 2 4 0

2 1 5 0

3 1 3 2

iiu(i,xu(i,x11) - ) -

ppii

u(i,xu(i,x22) - ) - ppii

u(i,xu(i,x33) - ) - ppii

1 0 0 -1

2 -1 1 -1

3 -1 -1 1

In order to search for a solution, we can search for:

•a competitive equilibrium and the corresponding price vector

CS483

An auction-based solution

Start with no objects allocated

Each agent bids for the object that maximizes utility - cost

Update the price of an object by the bidding amount

Terminate once the algorithm has found a feasible solution

S ← 0∀j pj ← 0repeat i: an unassigned agent // agent that will bid j ← argmaxk|(i,k)∈M (u(i,k) -pk) // object with maximum payoff bi ← (u(i,j) - pj) - argmaxk|(i,k)∈M;k≠j (u(i,k) -pk) // bi: the difference between the best object and the 2nd best object S ← add assignment (i,j) if there is another pair (i’,j) then remove it from S pj += bi

until S is feasible

CS483

Example

The auction-based algorithm will only terminate at a competitive equilibrium

But it doesn’t always terminate:

•when more than 1 object offers maximal value for a given agent

•and the objects happen to be the best for another agent

roundround pp11 pp22 pp33 bidderbidderpreferrpreferr

ed ed objectobject

assignmentassignment

ii u(i,xu(i,x11)) u(i,xu(i,x22)) u(i,xu(i,x33))

1 1 1 0

2 1 1 0

3 1 1 0

ii u(i,xu(i,x11)) u(i,xu(i,x22)) u(i,xu(i,x33))

1 2 4 0

2 1 5 0

3 1 3 2

0 0 0 0 1 x2 (1,x2)

1 0 2 0 2 x2 (2,x2)

2 0 4 0 3 x3 (2,x2),(3,x3)

3 0 4 1 1 x1(2,x2), (3,x3),

(1,x1)

CS483

A terminating auction algorithm

Idea:

•each time an agent makes a bid, add a small increment ε

-bi = u(i,j) - maxk|(i,k)∈M;k≠j u(i,k)+ ε

roundround pp11 pp22 pp33 bidderbidderpreferrpreferr

ed ed objectobject

assignmentassignment

0 ε 0 0 1 x1 (1,x1)

1 ε 2ε 0 2 x2 (1,x1), (2,x2)

2 3ε 2ε 0 3 x1 (3,x1),(2,x2)

3 3ε 4ε 0 1 x2 (3,x1), (1,x2)

4 5ε 4ε 0 2 x1 (2,x1), (1,x2)

iiu(i,xu(i,x11

))u(i,xu(i,x22

))u(i,xu(i,x33

))

1 1 1 0

2 1 1 0

3 1 1 0

CS483

ε-Competitive Equilibrium

The algorithm terminates

•if an object receives a bid in k iterations, its price must exceed its initial price by at least kε

•for sufficiently large k, the object will become expensive enough to be judged inferior to some object that has not received a bid so far

•once all objects receive at least one bid, the auction terminates

•worst case performance is O(n3k) for ε=1/n

•the smaller the ε - the worse the computational performance

But the competitive equilibrium property is no longer maintained

•agents overbid on some objects

ε-competitive equilibrium:

•S and p satisfy an ε-competitive equilibrium when for each i ∈ N, if

there exists a pair (i,j) ∈ S, then ∀ k: u(i,j) + ε ≥ u(i,k)

•A feasible solution S with n objects that forms an ε-competitive equilibrium with some price vector is within nε of optimal

•the smaller the ε - the best the approximation to the optimal solution

CS483

2. Scheduling Problems

A tuple C = (N,X,q,u)

•N is a set of n agents

•X is a set of discrete and consecutive time slots

•q = (q1,…, qm) is a price vector, where qj is the value that must be paid to reserve time slot xj

•u = (u1, …, un), where ui is the valuation function of agent i with 2 arguments:

- di: the deadline of agent i

- λi: the required number of time slots required by agent i

A solution to a scheduling problem is a vector F = (F0, F1, …, Fn), where Fi is the set of time slots assigned to agent i [F0: unassigned time slots]

So for an allocation Fi ⊂ 2X we have that:

•ui(Fi) = wi - if Fi includes λi hours before di;

•ui(Fi) = 0 - otherwise

Value of solution: V(F) = ∑j|xj∈F0 qj + ∑i∈N ui(Fi)

CS483

Scheduling example

Eight one hour slots: from 9:00am to 5:00pm

Reserve price of $3 per hour

jobjob lengthlengthλλ

deadlindeadlineedd

worthworthww

1 2 hours1:00p

m$10.00

2 2 hours12:00p

m$16.00

3 1 hour12:00p

m$6.00

4 4 hours5:00p

m$14.50

time slottime slot agentagent

9:00 am 2

10:00 am 2

11:00 am 1

12:00 pm 1

13:00 pm 4

14:00 pm 4

15:00 pm 4

16:00 pm 4

Integer Program Formulation:maximize ∑S⊆X,i∈N ui(S)xi,S

subject to ∑S⊆X xi,S ≤ 1 ∀i∈N ∑S⊆X:j∈S,i∈N xi,S ≤ 1 ∀j∈Xxi,S ∈ {0,1} ∀ S ⊆ X, i ∈ N

CS483

Generalized form of competitive equilibrium

Given a scheduling problem, a solution F is in competitive equilibrium at prices p if and only if:

•∀ i ∈ N: F = argmaxT⊆X (ui(T) - ∑j|xj∈T pj)

- the set of time slots allocated to agent i maximizes his surplus at prices p

•∀ j s.t. xj ∈ F0: pj = qj

- the price of all unallocated time slots is the reserve price

•∀ j s.t. xj ∉ F0: pj ≥ qj

- the price of all allocated time slots is greater than the reserve price

If a solution F to a scheduling problem C is in equilibrium at prices p, then F is also optimal for C

But...

•the scheduling problem is a hard problem

•the existence of a (generalized) competitive equilibrium cannot be guaranteed

CS483

Ascending-Auction Algorithm

Basic process:

•an auctioneer advertises an ask price

•agents bid the ask price for bundles of time slots that maximize their surplus at the given prices

•the process repeats until there is no change

Bid price vector: b = (b1, …, bm)

•bj: the highest bid so far for time slot xj

Allocated slots for each agent: F = (F1, …, Fn)

Difference with auction-based algorithm for the assignment problem:

•the bid increment here is always constant

•and does not depend on the difference between the best and the second-best assignment

Let ε be the allowed price increment

CS483

Ascending-Auction Algorithm∀ xj: bj ← qj // the initial bids are set to the reserve price∀ i: Fi ← 0repeat foreach agent i = 1 to N do foreach slot xj do if xj ∈ Fj then pj ← bj // agent gets the slot if the highest bidder else pj ← bj + ε //otherwise should increase bid S* ← argmax S⊆X|S⊇Fi (ui(S) - ∑j∈S pj)// find the best subset of slots, given outstanding bids// agent i becomes the high bidder for all slots in S* \ Fi foreach slot xj ∈ S* \ Fi do bj ← bj + ε // update bidding price if there exists agent k ≠ i so that xj ∈ Fk then set Fk ← Fk \ {xj} // update current allocations Fi ← S*until F does not change

CS483

Example

Assume an increment:

•ε = $0.25

round

bidder

slots bid on

F=(F1,F2,F3,F4) b

0 1 ( 9, 10)({9,10},{0},{0},

{0})(3.25,3.25,3,3,3,3,3,

3)

1 2 ( 10, 11)({9},{10,11},

{0},{0})(3.25,3.5,3.25,3,3,3,

3,3)

2 3 ( 09)({0},{10,11},

{9},{0})(3.5,3.5,3.25,3,3,3,3

,3)

⠇ ⠇ ⠇ ⠇ ⠇

24 1 0({11,12},{9,10},

{0},{12,13,14,15})

(6.25,6.25,6.25,3.25,3.25,3.25,3.25,3.25

)

jobjob lengthlengthλλ

deadlindeadlineedd

worthworthww

1 2 hours1:00p

m$10.00

2 2 hours12:00p

m$16.00

3 1 hour12:00p

m$6.00

4 4 hours5:00p

m$14.50

CS483

Properties

•The same algorithm for an increment of $1.00 does not reach the competitive equilibrium

•Even for small increments, however, there is no guarantee of convergence

•Unfortunately, even when the algorithm converges... it does not necessarily converge to to an optimal solution. We cannot even provide a bound!

•But the algorithm terminates!

- the prices of slots always increase

- so at some point unassigned slots will be also selected

jobjob length length λλ deadline ddeadline d worth wworth w

1 1 hour 11:00am $2.00

2 2 hours 12:00pm $20.00

3 2 hours 12:00pm $8.00

3 time slots9am,10am,11am

reserve price: $0

jobjob length length λλ deadline ddeadline d worth wworth w

1 1 hour 10:00am $3.00

2 2 hours 11:00am $11.00

2 time slots9am,10am

reserve prices: $1, $9