1

Self-Tuning Service Provisioning for DecentralizedCloud Applications

Raul Landa, Marinos Charalambides, Richard G. Clegg, David Griffin, and Miguel Rio

Abstract—Cloud computing has revolutionized service deliveryby providing on-demand invocation and elasticity. To reap thesebenefits, computation has been displaced from client devicesand into data centers. This partial centralization is undesirablefor applications that have stringent locality requirements, forexample low latency. This problem could be addressed with largenumbers of smaller cloud resources closer to users. However,as cloud computing diffuses from within data centers and intothe network, there will be a need for cloud resource allocationalgorithms that operate on resource-constrained computationalunits that serve localized subsets of customers. In this paperwe present a mechanism for service provisioning in distributedclouds where applications compete for resources. The mechanismoperates by enabling execution zones to assign resources basedon Vickrey auctions, and provides high-quality probabilisticmodels that applications can use to predict the outcomes of suchauctions. This allows applications to use knowledge of the localitydistribution of their clients to accurately select the number of bidsto be sent to each execution zone and their value. The proposedmechanism is highly scalable, efficient and validated by extensivesimulations.

Index Terms—Cloud Resource Management, DecentralizedCloud Applications, Vickrey Auctions, Quality of Experience.

I. INTRODUCTION

Cloud computing has been very successful, delivering appli-cations and services in a scalable manner through on-demandcomputation. It has allowed applications to elastically copewith changing demand for computing resources, and to exploiteconomies of scale in multi-tenancy data centers. However,this has meant that in many cases execution points can onlybe selected from a limited number of options; this couldpotentially reduce the quality of experience (QoE) provided tothe customers, or increase the bandwith requirements betweengeographically dispersed customers and cloud data centers.

Many resource-demanding services (including personalizedreal-time video, games or processing of high bandwidthstreams for security, safety and health-care monitoring) arenot suited to being centralized in a relatively small number ofdata centers where higher network delays and low throughputcan have a serious impact on the QoE experienced by manyusers. Application providers do not normally have the meansto position cloud application logic very close to their users inorder to meet tight quality constraints or to avoid congestedor expensive network paths for high bandwidth flows.

There is a current trend for deploying cloud resources closerto users as telcos and ISPs deploy data centers at the networkedge. In addition, routers and other network equipment can

The authors are with the Department of Electronic and Electrical Engineer-ing, University College London, UK.

now also perform general-purpose computation for virtualizednetwork functions; these can be deployed by ISPs themselvesor offered to third party providers to deploy applications andservices [1]. Such a distributed cloud drastically reduces RTTto customers and greatly relaxes the end-to-end bottleneckbandwidth requirements between end users and these localizedcloud points of presence. Distributed cloud resources can bedeployed in various places throughout the Internet: in network-edge points of presence close to users, as an extension totypical CDNs; in specialized data centers owned and operatedby ISPs; and in traditional data centers and service farmsoperated by cloud providers. Unfortunately, each one of thesesmall cloud execution points will be unable to provide theessentially boundless elasticity that purpose-built data centerscan. This means that, in many instances, there can be morerequests for resources at a given execution point than thereare resources available. In these cases, resource allocationmechanisms that deal with service contention will be required.

In this paper we present a resource allocation mecha-nism for service provisioning in distributed clouds wherecomputation resources are made available much closer tousers, possibly within routers themselves. This will savebandwidth, provide better user QoE and potentially complywith regulatory frameworks that demand services to run inspecific regions/countries/states. Applications can then usethese resources to deploy demanding services. Of course,when securing resources, applications will need to balance thebenefit that these resources bring to the QoE of their users withthe costs that they need to pay to the computation resourceproviders. To this end, we propose an auction-based resourceallocation approach that allows cloud providers to providecomputation capacity in a decentralized manner, while at thesame time allowing applications to implement effective cost-benefit tradeoffs. We take as given the well-known propertiesof the Vickrey auction (eg. truthful revelation and ex-postefficiency [2]) and focus on their use as a vehicle to imple-ment a self-adaptive mechanism for the dynamic allocationof service slots, obviating the need of a logically centralizedclearinghouse.

In the paper we show that the proposed approach providesan effective option to deal with contention and oversubscrip-tion in decentralized cloud scenarios. We prove that it isboth scalable and efficient, but also immune to false-nameattacks. We present both the analytical work that underpinsour design and an extensive set of simulations that demonstrateits behavior in many cases of interest. The dynamic behaviorof the algorithm has been investigated using three differentconvergence approaches and it is shown that the algorithm

2

converges to the same solutions.The paper is organized as follows. In Section II we describe

the main actors of our proposed mechanism. We continuein Section III by explaining the analytical foundations ofour resource allocation mechanism. In Section IV we presentthe results of evaluating our proposed mechanisms throughsimulation in a wide variety of situations. Finally, we presentan overview of the related work in auction-based resourceallocation in Section V and our conclusions in Section VI.

II. SYSTEM OVERVIEW

In our proposed system we assume the existence of alarge number of execution zones (EZ), where computationalcapabilities are offered to applications. Resources in each EZare managed by a zone manager (ZM), which is operated bythe infrastructure provider. Applications can deploy services(i.e. self-contained application logic components) on these EZsunder the direction of an application manager (AM), which isoperated by the application provider and implements the ap-propriate cost-benefit tradeoffs for that application. The AMsbelonging to various applications then compete for resourceson the EZs. Rather than relying on a centralized resourceallocator to grant AMs access to EZ resources, we alloweach EZ to individually perform resource allocation decisions.There is a rich literature on how to solve this problem [3]–[5];we propose the use of a market system to allow EZs to allocateresources to those applications that value them the most (see[6]–[9]). We take advantage of the well known properties ofVickrey auctions [2] to allocate resources to applications (see[10]–[15] for other examples of auctions used for resourceallocation in grid, peer-to-peer and cloud computing). Ourmechanism allows each AM to bid for resources on the desiredEZs according to its individual policies, and to implement itsown tradeoff between user satisfaction and infrastructure cost.This involves considering the monetary cost of the resourcestogether with the cost of users being blocked due to resourceunavailablity.

We consider latency- and bandwidth-sensitive serviceswhich require accurate placement of service instances so thatthe network performance metrics between users and cloud-based services are within acceptable bounds to ensure QoE.There are two dimensions to QoE in our model. Firstly there isthe quality experienced by the users of an application in termsof the latency, bandwidth and response time of a session. Thisis determined by an AM selecting EZs that are topologicallyclose to the users in terms of network metrics, such as latency,and by the EZ offering sufficient computational capabilities(e.g. CPU, memory) for the application. This aspect of QoE isdetermined by decisions made by the AM to map user demandto appropriate EZs and is part of the AM logic outside of theauction mechanism described in this paper. The outcome ofthis decision determines the demand on a specific EZ andhence will drive the quantity of bids an AM should make to aspecific EZ. The second dimension of QoE, termed QoS cost,is related to the blocking probability of user session requestsat an EZ. Our model deals with both dimensions of QoE,but only the blocking probability is directly impacted by the

Zone%Manager%

Zone%Manager%

Cloud%Resources%%%%

Service%Load%Balancer%

Service%Instance%

Service%Instance%

Applica:on%Manager%

Cloud%Resources%%%%

Service%Load%Balancer%

Service%Instance%

Execu:on%Zone% Execu:on%Zone%

Service%Queries%

Bids% Bids%

Bidding/Demand%Informa:on%

Bidding/Demand%Informa:on%

Fig. 1. Notional architecture

auction outcome. Latency, throughput and other metrics of anactive session are indirectly covered by the prior selection ofthe EZs where an AM will place bids for service slots. For thisreason, and in order to maintain our architectural decouplingbetween AM and EZ functions, we assume that bids fromAMs are relevant only for one specific EZ. Off-loading ofdemand from one EZ to another may violate QoE constraintsand the placement decisions made by the AMs. Of course, acloud provider with multiple physical data centers within aregion may act as a single EZ if the underlying locations areequivalent in terms of latency, bandwidth and other networkperformance metrics.

We focus our attention on the dynamic use of EZ resourcesby AM. To simplify the resource allocation tasks of EZs, wewill assume that each EZ will offer a discrete number ofidentical service slots. Rather than expressing service execu-tion requirements in terms of CPU speed, amount of memory,etc. EZs abstract the underlying computational resources asservice slots. Cloud providers, such as Amazon EC2, offermultiple virtual computing instance types that vary in termsof the processing and storage capabilities of the underlyinghardware resources. We assume that an AM has identified theresource type offered by a data center that is best suited totheir applications, e.g. to optimize compute power, storage ormemory, or the use of specialized resources such as GPUs.A service slot for an application will be mapped to a singleinstance type and an AM will therefore bid on only one typeof instance. Our model assumes that a cloud provider offeringmultiple instance types could operate multiple, collocated EZs,one for each instance type and therefore a separate auctionwhere AMs offering applications with similar requirementswill compete for resources. We assume that prior negotiationhas established the type and quantity of resources required tosupport a service slot for the applications being deployed bythe AM. In addition to its scalability benefits (see SectionIV-C), this abstraction simplifies the resource modeling inEZs and allows a more straightforward calculation of thetradeoffs involved. Our model therefore allows heterogeneousapplication requirements and heterogeneous resources to beaccommodated without introducing the complexity of a multi-parameter auction.

According to the definitions above, our proposed mecha-nism, which is depicted in Fig. 1, can be explained as follows.

3

AMs determine which EZs are capable on computationalgrounds for hosting their application. From this subset of EZsan AM will determine how demand from their users shouldbe mapped to EZs so that network metrics, such as latencyand bandwidth, do not violate application QoE requirements.Users will typically be distributed over a wide area and a singleEZ is unlikely to be positioned to achieve the required QoEto all user locations. Hence, our model assumes that an AMwill select multiple EZs to deploy the application and willbid for resources in all EZs it has selected (the quantity ofservice slots being determined at each EZ by the user demandin that location). Application users generate service queries,which are directed to their local EZs by existing methodslike differentiated DNS resolution and HTTP redirection, asconfigured by the application provider. These queries arereceived by ZMs and mapped to the service slots that theapplication has available on that zone. Each AM can secure anumber of service slots for its application in each EZ; this isachieved by sending bids to the ZMs specifying the number ofservice slots desired and the bid amount that they are willing topay per service slot. The ZM then allocates its available serviceslots by running a multi-item Vickrey auction [16]; the topbidding AM bids are hence selected. AMs make a centralizeddecision on which EZs should host their application to meetQoE requirements and on the quantity of resources requiredto meet the demand at that EZ (see Fig. 1). Each EZ runs anauction for resources with bids being generated by the set ofAMs that have selected the EZ. There is a set of independentauctions, one at each EZ (see Fig. 3), where each auction is forlocal resources only and therefore no coordination is requiredbetween independent auctions.

Among alternatives, eg. open ascending / descending bidauctions, we have chosen to use Vickrey auctions due totheir low overhead, which preserves system scalability. Fur-thermore, compared to other sealed bid auction types, eg. firstprice, in Vickrey auctions, true value bidding is the dominantstrategy (truthful revelation). Since the Vickrey auction isa VCG mechanism [2], it is ex-post efficient: it allocatesresources in a way that is optimal for the entire system. Thisis the reason it is frequently used as a protocol component inself-organising resource allocation [2] [17].

III. AUCTION-BASED RESOURCE ALLOCATION FOR THEDECENTRALIZED CLOUD

We assume that the AM can determine, by external means,the total user demand associated with a given EZ. Then, foreach AM the resource allocation problem becomes: given thebidding profile at every EZ, for how many service resourceunits should the AM bid, and at what price? Formally, eachAM bj must determine the number of service slots mij it willbid for at each EZ j,as well as the bid price vij it must offer,according to the estimated user demand and expected servicequality.

We now present our analytic model of the EZ, and howit implements resource allocation using Vickrey auctions. Ina standard Vickrey second price auction the bidders bid fora single item. The highest bid wins and pays the price of

the second highest bid. This can be shown (under certainassumptions) to mean that rational bidders will bid accordingto their true valuation for the item. In a multiunit Vickreyauction with identical items, the bidders bid for K items andmay each bid multiple times. The highest K bids win andthey pay the combined value of the next K bids (the first Klosing bids). This preserves the property that rational biddersbid according to how much they value the item.

In the context of this paper, we assume that each EZ ni canoffer up to Mi service slots (see Table I), and that each AMbj sends mij bids to ni, each at a value vij . The EZ receivesthese bids and ranks them in increasing order of value, andthe top Mi win a service slot. Let Xij be the number of bidsthat AM bj wins in EZ ni. According to the definition of aVickrey auction, the payment that will be levied on managerbj is the sum of the top Xij losing bids.

By the construction of a Vickrey auction the value of thebids that ni makes in bj are a true reflection of the benefit itwould gain from those bids if they were successful. Therefore,the utility Uij that bj obtains from this interaction with niis just the difference between the sum of the values of itsXij winning bids and the sum of the top Xij losing bids (itspayment to the EZ). The outcome of this process is that eachAM calculates the price of its bids to each EZ according tothe value it expects to derive from service slots located in thatEZ.

A. Resource Allocation Properties

Even though the Vickrey auction has interesting advantages(e.g. incentive compatibility and ex-post efficiency), it is notrevenue-maximizing [2] [17]. In fact, for uncontested Vickreyauctions, the bidders can obtain items at zero cost. For thepresent paper we assume that AMs pay a monthly fixed feeto gain access to the EZ infrastructure; this would allow themto use uncontested resources. If AMs wish to gain access toEZ resources for which there is contention, they will need tosubmit nonzero bids and pay for these bids in addition to thefixed flat fee. In contrast to other contributions in the literature,we assume that bids are made in terms of actual currency andthe EZs receive income from both the monthly rental fees andnon-zero winning bids from AMs.

We base our auction mechanism on two simplifying assump-tions. First, we will use the independent private values modelin the sale of Mi service slots to Ni managers. This meansthat the value that each AM bj gives to service slots auctionedby an EZ ni is private, and is not a function of the values thatother AMs give to service slots offered by ni. This simplifiesthe valuation problem, decoupling the decisions for each AM.Second, we will assume that AMs experience no synergy inacquiring increasing numbers of service slots. This means thatfor each manager bj , the value of obtaining Xij service slotsfrom ni is just Xij times the value of obtaining a singleservice slot from ni. Therefore, there are no complementarityor substitution effects between different service slots, and allservice slots from a given EZ have the same value.

In addition to making the problem mathematically tractable,these two assumptions have an additional benefit: they make

4

our proposed mechanism false-name proof [18] [19]. In acombinatorial auction bidders bid for sets of non-identicalitems, and they have independent valuations for each one ofthese sets. This makes the system vulnerable to false-nameattcks, in which bidders can use additional identities to submitstrategic bids that allow them to obtain items at a reducedcost. Although false-name proofness is in general impossibleusing Vickrey auctions [18], our proposed mechanism avoidsthis problem by mandating service slots within each EZto be identical (for combinatorial auctions, different auctionmechanisms can regain the false-name-proof property [19]).

Theorem 1. In a system where service slots are allocated toAMs following the mechanism described in Section III-A, thereis no incentive for AMs to submit bids at any valuation otherthan their true valuation.

Proof. The false-name attack is geared towards obtaining thesame set of items in the auction but at a lower cost; hence, ouranalysis assumes that the bidder will only bid for its desirednumber of service slots. Further, since bidders can only specifythe number of slots that they bid for and the unitary valuationthat they have for each one, we only need to consider threepossibilities: that the bid is equal, lower or higher than theirtrue value.

1) False-name bids are made at the same valuationas true-name bids. This case is indistinguishable fromsimply submitting the same number of bids using asingle name. Hence, in this case the bidder has noincentive to use an alternate identity.

2) False-name bids are made at a lower valuation thantrue-name bids. We can distinguish two cases here. Iffalse-name bids are low enough so that the bidder failsto secure service slots that it would have secured byrevealing its true valuation, the bidder will be forced toforgo service slots which were available at a price thatthe bidder was willing to pay. Consequently, it will haveexperienced a lower utility than it would have had it bidits true valuation. On the other hand, if the false-bidsare lower than its true valuation, but still high enoughto secure all required service slots, the amount paidis identical to that obtained by bidding normally usingits true name. Hence, under-bidding with an alternativeidentity can not increase its utility, and may potentiallydecrease it.

3) False-name bids are made at a higher valuation thantrue-name bids. Again, we can distinguish two cases. Iffalse-name bids are not high enough so that the bidderwins service slots that it would not have won by biddingits true valuation, its utility is unchanged. If false-namebids are high enough to make the bidder win serviceslots that it would not have won by bidding its truevaluation, its utility will be negative for those serviceslots. To see why, consider that the true valuation forthose service slots is necessarily lower than that of thelowest bid in the winning set (by the definition of ourauction mechanism in Section III-B). When these bidsdisplace a number of previously winning bids, these willbecome the highest losing bids, and hence, will define

the price that the bidder will have to pay. However, thisprice is higher than its true valuation, and hence, willlead to a negative utility. It follows that over-biddingwith an alternative identity can not increase its utility,and may potentially decrease it.

B. Estimating the Expected Number of Successful Bids

To allow an AM to estimate the expected number of bids itwill win and the total price it will pay when placing a givennumber of bids at a given price, each with a given ZM, it needsto estimate the level of competition it will face in the auctionfrom other AMs. Formally, this is achieved by determiningthe number of competing bids it will encounter. Thus, ratherthan the total number of bids Ni received by ni, an AM bjis interested in the number of bids with which its bids mustcompete. This involves subtracting from Ni those bids sent toni by bj itself. We shall denote this number of competing bidsas N j

i , the total competing load of manager bj at EZ ni. Bydefinition, N j

i =∑k 6=imkj .

We now define two basic load-related states for the EZ. Wecall the case where Mi > N j

i the surplus state, and the casewhere Mi ≤ N j

i as the scarcity state. In the surplus statethere are more service slots in auction than bids submitted bycompeting AMs, and bj can expect to have some of its bidsaccepted at zero price. Winning bids pay nonzero prices onlyin the scarcity state, when bj competes with other managersand there are losing bids.

It will be advantageous for AMs to have a model of theexpected auction outcome after bidding with a given EZ. Tothis end, we investigate how an AM bj can predict, given ahypothetical bidding value vij , the expected number of bids itwill win when submitting bids to a particular ZM ni. Since bythe truthful revelation property of the Vickrey auction [2], [17]bidders have no incentive to bid strategically, we can safelyassume that vij is the true value that every AM estimates forservice slots being auctioned by ni. In what follows, we shallrefer to a given execution manager ni or its associated EZinterchangeably; likewise, we will denote as bj both the AMand the application itself.

We simplify our analysis by modeling the bid values asa random variable V i with a probability density fV

i

andcumulative distribution FV

i

. Then, FVi

(vb) − FVi

(va) =∫ vbvafV

i

(ω)dω gives the proportion of values between va andvb for all those bids that were sent to ni from all AMs. SinceAMs do not bid strategically, EZs can compile these valueprobability distributions simply by performing elementaryfrequency counts (histograms) over their received bids.

Again focusing on the prospective competition for AM bj ,in the model presented here we treat the N j

i competing bidsthat ni receives in a given interval T as a set of N j

i realizationsof the random variable V i. We denote this vector of variatesdrawn from fV

i

as V i = {V i1, V i2, . . . , V iNji−1

, V iNji}. To

model a bidding round in a Vickrey auction, we assume thatthe EZ ni constructs V i using the bid values it receives for thatround. Then, it sorts V i in ascending order, creating V ij =

5

TABLE IAUCTION-BASED RESOURCE ALLOCATION MODEL NOTATION

Ni Total number of bids that zone manager ni receivesNj

i Total number of bids that zone manager ni receives whichcompete with those placed by bj

Mi Total number of service slots offered in execution zoneni

mij Total number of bids that application manager bj sendsto zone manager ni

vij Monetary value for each bid that application manager bjsends to to zone manager ni

Xij Number of bids that bj wins on ni

Xijfree Number of contention free service slots (ie. obtainable

at zero auction cost) that bj can obtain when submittingbids to ni

Xijmax Maximum number of bids that bj can win by bidding

with ni

fVi(vij) Probability density function of bid values received by

auctioneer ni

FV i(vij) Cumulative distribution function of bid values received

by auctioneer ni

fVi

(k)(vij) Probability density function of the k-th order statistic of

the bid values received by auctioneer ni

FV i

(k)(vij) Cumulative distribution function of the k-th order statistic

of the bid values received by auctioneer ni

φij Maximum losing rank in the Vickrey auction; biddinground after which bids sent to ni by bj will necessarilybe lost

µij Expected number of successful bids that bj will havewhen bidding with ni

E [Cij ] Expected monetary cost that bj will incur when submit-ting mij bids to ni at a bid price vij

E [Qij ] Expected QoS cost that bj will incur when submittingmij bids to ni at a bid price vij

E [Tij ] Expected total cost (monetary and QoS) that bj will incurwhen submitting mij bids to ni at a bid price vij

{V i(1), V i(2), . . . , V i(Nji−1)

, V i(Nji )}, keeps the top Mi bids

in V ij , and discards the rest.

Theorem 2. Assume a Vickrey auction where applicationmanagers bid for execution slots within zones. Let vij be thevalue to application manager bj of winning one slot in zoneni and assume that the value of winning s such slots is svij .Let FV

i

(vij) be the CDF of the bid values received by nifrom bj . Let N j

i be the total number of bids received by niexcluding those placed by bj . Let φij be the maximum losingrank in the auction and let Xij

max be the maximum number ofbids that bj can win by bidding with ni. The expected numberof slots won by bj in zone ni is given by

µij =

Xijmax∑k=1

I(FVi

(vij) ; φij + k,N ji − (φij + k) + 1)., (1)

where I(x; a, b) is the regularized incomplete beta function.

Proof. We are interested in the probabilities of the variouspossible outcomes of the auction (see Figure 2). We considera situation where every AM in the system but bj has alreadysubmitted all their bids to ni, so that the N j

i , Mi and mij

are set. In this case, since N ji is the total number of bids that

ni will receive in addition to the mij that bj will send, wehave that Ni = N j

i +mij . If N ji ≥ Mi, ni is in the scarcity

state, and bids from bj will face competition. In particular, wesee that the lowest bid that bj will have to surpass will be the

(N ji −Mi+1)-th highest bid received by ni, which is the last

winning bid in its absence.On the other hand, if Mi > N j

i , ni is in the surplus state andbj will have access to Xij

free = Mi−N ji contention-free service

slots: the first Xijfree bids from bj will win without competition

and have zero cost. Any bids following, however, will facethe same competition situation detailed above, although in thiscase with a reduced effective number of bids m′ij = mij −Xij

free = mij−Mi+N ji . Thus, we see that if mij < Xij

free, theAM wins all its bids at zero cost, and if mij ≥ Xij

free, Xijfree of

its bids win at zero cost and m′ij face the EZ in the scarcitystate. Without loss of generality, we focus on the scarcity stateand assume that N j

i > Mi.We see in Figure 2 that bj will win no bids with a probability

P[vij < V i(Nji−Mi+1)], and it will win exactly one bid with

probability P[V i(Nji−Mi+1) < vij < V i(Nj

i−Mi+2)]. Thispattern continues until either all the bids that were submittedto the EZ have been considered (the case shown in Figure2) or all the service slots that ni has in auction have beenconsidered. This bounds Xij , the maximum number of bidsthat an AM can win, to either the total number of units inauction Mi or the number of bids that bj submits, mij .

Thus, we have that Xijmax = min(mij ,Mi), and this can

happen with a probability of P[V i(Nj

i−Mi+Xijmax)

< vij ]. Forµij , the expected number of successful bids that bj will haveper auction, we have that

µij = E[Xij ] =

Xijmax−1∑k=1

k P[V i(φij+k)< vij < V i(φij+k+1)]

(2)

+XijmaxP[V i

(φij+Xijmax)

< vij ],

where φij = N ji − Mi. Since we have that P[V i(φij+k) <

vij < V i(φij+k+1)] = P[V i(φij+k) < vij ]− P[V i(φij+k+1) <vij ], we see that the expression for µij telescopes and can betrivially simplified. Thus, it follows that

µij = E[Xij ] =

Xijmax∑k=1

P[V i(φij+k) < vij ]. (3)

We find P[V i(φij+k) < vij ] by noting that it correspondsto the (φij + k)-th order statistic of the probability densityfV

i

of bid values [20], [21]. Formally, order statistics can bedefined as follows.

Definition 1. Let V i ≥ 0 denote a continuous randomvariable with probability density fV

i

(v), and cumulative dis-tribution FV

i

(v) =∫ v0fV

i

(ω)dω. Let (V i(1), Vi(2), . . . , V

i(Nj

i ))

denote a random sample of size N ji drawn on V i, and ordered

so that V i(1)<Vi(2)<. . .<V

i(Nj

i ). Then, the V i(1), V

i(2), . . . , V

i(Nj

i )

are collectively known as the order statistics derived from V i.The probability density for the k-th order statistic will bedenoted as fV

i

(k)(v), and can be calculated [21] as

fVi

(k)(v) =N ji !FV

i

(v)k−1(1− FV i

(v))Nji−kfV

i

(v)

(k − 1)!(N ji − k)!

. (4)

6

ProbabilityBidsWon

BidRank

WinnerRank

Ni

Ni ! Mi + mji

Ni − Mi + 1

Ni − Mi + 2

Mi

mji

1

2

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

mij

1

0

P[V i(Ni)

! vij ]

P[V i(Ni!Mi+mji)

! vij ! Vi(Ni)

]

P[V i(Ni!Mi+2) ! vij ! V

i(Ni!Mi+mji)

]

P[V i(Ni!Mi+1) ! vij ! V

i(Ni!Mi+2)]

P[vij ! Vi(Ni!Mi+1)]

Fig. 2. Calculating the expected number of won bids

We denote the cumulative distribution of the (φij + k)-thorder statistic of V i, as FV

i

(φij+k)(vij), so that

P[V i(φij+k) ≤ vij ] = FVi

(φij+k)(vij) =

∫ vij

0

fVi

(φij+k)(ω)dω,

(5)where fV

i

(φij+k)(ω) denotes the probability density for the

(φij + k)-th value in our sorted vector V ij , and FVi

(φij+k)(vij)

denotes its cumulative distribution. By substituting the defini-tion for fV

i

(φij+k)(ω) in (5) and simplifying, we find that

FVi

(φij+k)(vij) =

B(FVi

(vij) ; φij + k,N ji − (φij + k) + 1)

B(φij + k,N ji − (φij + k) + 1)

,

where FVi

(vij) is the cumulative density of the bid valuessubmitted to ni valuated at vij , and B(φij + k,N j

i − (φij +k)+1) and B(x ; φij+k,N j

i −(φij+k)+1) are, respectively,the beta function B(a, b) and the incomplete beta functionB(x ; a, b) with parameters a = φij + k and b = N j

i − (φij +k) + 1. Formally, these are defined as

B(x ; a, b) =

∫ x

0

ωa−1(1− ω)b−1dω [0 ≤ x ≤ 1]

B(a, b) = B(1 ; a, b) =

∫ 1

0

ωa−1(1− ω)b−1dω.

We can further simplify (5) by using I(x ; a, b), the regular-ized incomplete beta function with parameters a and b, whichis defined as

I(x ; a, b) =B(x ; a, b)

B(a, b)[0 ≤ x ≤ 1].

We can see then that

P[V i(φij+k) ≤ vij ] = I(FVi

(vij) ; φij+k,Nji −(φij+k)+1),

where I(x ; φij + k,N ji − (φij + k) + 1) is I(x ; a, b) with

parameters a = φij + k and b = N ji − (φij + k) + 1. By

substituting this last expression in (3), we then have (1) fromthe theorem.

µij =

Xijmax∑k=1

I(FVi

(vij) ; φij + k,N ji − (φij + k) + 1).

Since I(0; a, b) = 0 and I(1 ; a, b) = 1, we have thatlimvij→0 µij = 0 and limvij→∞ µij = Xij

max, as expected:in the first case no bids are won, while in the second onethe maximum number of feasible service slots are won. Thiscompletes the proof.

C. The Expected Resource Cost

We now calculate the expected cost of an AM when biddingat a given EZ.

Theorem 3. Given the auction mechanism from theorem 2 theexpected cost paid by an AM bj to a ZM ni is given by

E [Cij ] = vijµij −∫ vij

0

µij(ω)dω. (6)

Proof. This cost, for each won bid, equals the price that bjneeds to pay to ni. Then, for the cost Cij paid by bj if itsends mij bids to ni, we have for N ≤ Xij

max won bids

Cij =

N∑k=1

V i(φij+k)if V i(φij+N) < vij

and thus we have for the expected cost E[Cij ] that

E[Cij ] =

Xijmax∑k=1

∫ vij

0

ωfVi

(φij+k)(ω)dω,

where, again, fVi

(φij+k)(ω) is the probability density of the

(φij + k)-th order statistic of V i. The integral within thesummation can be partially evaluated and simplified usingintegration by parts, and we find that∫ vij

0

ωfVi

(φij+k)(ω)dω = ωFV

i

(φij+k)(ω)∣∣∣vij0

−∫ vij

0

FVi

(φij+k)(ω)dω.

Hence, we have that

E[Cij ] =

Xijmax∑k=1

[vijF

V i

(φij+k)(vij)−

∫ vij

0

FVi

(φij+k)(ω)dω

],

and therefore, if we exchange summation and integration, that

E[Cij ] = vij

Xijmax∑k=1

FVi

(φij+k)(vij)−

∫ vij

0

Xijmax∑k=1

FVi

(φij+k)(ω)dω.

Let µij(·) represent µij expressed as a function of the bidvalue vij and hence

E [Cij ] = vijµij −∫ vij

0

Xijmax∑k=1

FVi

(φij+k)(ω)dω

= vijµij −∫ vij

0

µij(ω)dω.

7

D. The Expected QoS Cost

In this section we consider the QoS cost that an AM expects.The QoS cost can be considered as a penalty to QoS that theapplication has, given the number of service slots secured.

Theorem 4. Given the auction mechanism from theorem 2define Qij as the QoS cost that users of application bjexperience in EZ ni. This is a function of the number of serviceslots secured. We define Qij(k) as the QoS cost from securingexactly k slots in ni. Then we have

E[Qij ] = Qij(0) +

Xijmax∑k=1

∆Qij(k) (7)

I(FVi

(vij) ; φij + k,N ji − (φij + k) + 1),

where ∆Qij(k) = Qij(k) − Qij(k − 1) is the QoS costdifference achieved by adding the kth service slot.

Proof. In this case, we assume that customer queries willreceive a QoS that is a function of the current number ofservice slots that an application has obtained in a given EZ.If we define Qij(k) as the QoS cost that users of applicationbj experience when using EZ ni (with k service slots securedby bj), we can calculate the expected QoS cost as

µQij = E[Qij ] = Qij(0)P[vij < V i(φij)]

+

Xijmax−1∑k=1

Qij(k)P[V i(φij+k)< vij < V i(φij+k+1)]

+Qij(Xijmax)P[V i

(φij+Xijmax)

< vij ].

By following a reasoning identical to (3), we see that

µQij = Qij(0) +

Xijmax∑k=1

∆Qij(k)P[V i(φij+k) < vij ].

The theorem follows immediately.

By comparing (1) and (7) we see that whereas k runs from1 in (1), it runs from 0 in (7). The reason for this is that,whereas k = 0 in (1) contributes nothing to µij , k = 0 in (7)has an important contribution to µQij because it represents thesituation where no client requests are served, and hence, themaximum QoS cost that can be incurred.

To understand the boundary properties of µQij , we assumethat the QoS cost Qij(k) decreases as the number of serviceslots k (bids won) increases. This implies that ∆Qij(k) willbe negative for all values of k. Hence, µQij attains its maximalvalue of Qij(0) for k = 0 and it then decreases with increasingk. In addition, we have that limvij→0 µ

Qij = Qij(0) and

limvij→∞ µqij = Qij(Xijmax), as expected.

E. Bid Determination: The Expected Total Cost

Suppose that an AM bj wishes to determine the number andvalue of bids that it must submit to a given execution managerni, and it wishes to do so taking into account both the resourcecost and client QoS. This can be achieved by selecting thecombination of mij and vij that minimize a total cost measurethat includes both QoS and monetary bidding effects. We use

the simplest possible definition, in which the total expectedcost is simply the sum of the resource and the QoS costs.However, since the resource cost is in monetary units and theQoS cost is in QoS units, we require a mapping between thesetwo units of measurement so that these costs can be added.We assume a linear mapping based on a static coefficient ζthat gives the monetary cost associated with a given QoS cost.This does not impose any loss in generality, as the nonlinearaspects of QoS cost will be considered separately: the role ofζ is only to allow the joint consideration of QoS and monetarycosts into a single model. Then, for this total cost Tij we have

E [Tij ] = E [Cij ] + ζE [Qij ] . (8)

However, in order for Tij to be fully defined, it is necessaryto define the quality cost Qij(k). To achieve this, we assumethat each EZ i will provide service to a given population ofcustomers that generate service requests at a rate of λij perunit time following a Poisson distribution. Further, we assumethat each service slot can process service requests at a rateof ri per unit time, with the service time being exponentiallydistributed. Finally, we assume that each service slot can onlyprocess one request at a time. The queueing model, thereforehas exponential interarrival times, exponential service times,N servers and at most N customers. In standard Kendallnotation [22], this is an M/M/N/N model.

We use the loss probability of the the M/M/N/N model todefine the QoS cost Qij(k). Intuitively, Qij(k) is defined asthe probability that a customer request will be denied service(i.e. blocked) from EZ ni, given a service request rate of λijand a service completion rate of ri. Formally, if we define theoffered load ρij =

λij

riand the full-blocking cost Bij , we have

Qij(k) =

ρkijk!∑k

n=0

ρnijn!

Bij . (9)

By using the recursive definition of the Erlang B loss proba-bility [22], we find that

∆Qij(k) =

[1−Qij(k − 1) +

k

ρi

]Qij(k), (10)

where the recursion terminates with Qij(0) = 1. This willallow our simulation engine to calculate Qij(k) efficiently.

With these definitions, we can formulate the bid planningproblem for each AM as

Minimize:mij∈N≥0,vij∈R≥0

E [Tij ] . (11)

This problem could be solved by using nonlinear mixed integerprogramming. In the case of this paper, however, the statespace is sufficiently small that it can be explored exhaustively.That is, each AM can try all combinations of mij , vij and seewhich values do minimize E [Tij ].

F. Stability

A common pitfall in decentralized optimization mecha-nisms, such as the one presented in this paper, is the possibilityfor oscillation (see eg. [23], [24]). We address this concernby using the optimal answers provided by (11) to guide an

8

incremental bidding strategy, thus dampening oscillations andensuring convergence. During each epoch, each AM calculatesthe optimal mij and vij based on the level of competition andresource availability in each EZ. Instead of using these directly,AMs use approximations mij and vij that are designed to varywithout risking large oscillations, meaning, free from largestep changes. We do this by limiting the change in mij andvij from one epoch to the next. Furthermore, we also ensurethat both mij and vij remain bounded by specifying respectivemaximal values mmax and vmax.

Three approaches are taken. The first approach is to movethe number of bids at most one unit towards the optimalnumber. The second is to move the number of bids only afraction of the way towards the new optimal and the third isto do this but reduce the fraction as time progresses.

Approach 1: This changes the number of bids by +1 (-1) ifit is too low (high) with probability p, and leaves it unchangedwith probability 1− p. We used p = .5 in our evaluation.

Approach 2: This changes the number of bids in a mannerbased on the exponentially weighted moving average. The bidis increased (or decreased) by an amount proportional to thedifference between the previous bid and the ideal bid. Thatis, the bid increases (or decreases) by w(mij − m′ij) wherem′ij is the previous bid and 0 < w < 1 is some constant. Thenearer w is to 0 the more conservative the system is. Since bidsmust be whole numbers then the fractional part is treated as aprobability, for example if w(mij −m′ij) = 1.2 then the bidwill increase by 1 (with probability p) or 2 (with probability1− p). We used p = .8 in our evaluation.

Approach 3: This is like Approach 2 but the factor w isdecreased each round by multiplying it by some constant 0 <d < 1. So, as epochs progress the changes to bids becomemore conservative.

It was found that the final results were similar in almost allcases investigated and the eventual position was insensitiveto the convergence approach used (although on occasion thetransient positions early in the simulation could be quitedifferent). This is discussed further in Section IV-D. Theresults presented in the paper are for Approach 2 with w = 0.1.

The following section presents the simulation set up and theresults of the evaluation of our proposed mechanism.

IV. EVALUATION

We evaluate our mechanism through simulation. The sim-ulation proceeds by epochs, with one epoch consisting of abidding phase in which AMs decide, independently and inparallel, how many bids to submit to each EZ and at whatvaluation. These values are accumulated at each EZ, and whenall required values are available, the system moves on to theallocation phase. At this point, EZs, independently and inparallel, perform resource allocation on the available bids andcalculate the QoS experienced by the users of each service intheir designated population. Statistics are then gathered andmade available to AMs so that they can adapt their biddingbehavior.

Applica'on  Manager  1  

Applica'on  Manager  2  

Applica'on  Manager  3   …  

Zone  Manager  1  

Zone  Manager  2  

Zone  Manager  3  

Zone  Manager  4  

Zone  Manager  5   …  

Fig. 3. Simulator architecture

A. Simulator Setup

Each application and ZM in our simulator is implemented asan independent thread and triggered by means of a thread pool[25]. As already stated, simulation progresses by discrete timesteps denoted as epochs. Each epoch consists of two steps,and each one of these steps is implemented as a map/reduceoperation. Simulation state is synchronized after each reducestep. We now broadly explain the two main steps of eachsimulation epoch by focusing in turn on the EZs and theAMs (see Fig. 3). The first step, denoted as the executionstep, implements functionality to allow EZs to receive bidsfrom all AMs, choose the winning set of bids, and calculatethe QoS experienced by application users. Each EZ can thenreport to each AM the total number of bids Ni it received, thetotal number of service slots Mi it offers, and the distributionFV

i

(vij) of its received bid values. The second step, denotedas the bid planning step, implements functionality to alloweach AM to evaluate (1), (6) and (7) as a function of mij andvij . Then, each AM can determine the optimal mij , vij pairthat minimizes its expected total cost as expressed by (8), anduse this information to drive its bidding behavior. A summaryof the main simulation steps is provided below.

1) Initialize simulation - generate EZs and AMs accordingto QoE tolerance and population density distributions

2) Execution stepa) AMs make bids and send to EZsb) Winning bids determined and results disseminated

3) Bid planning stepa) AMs calculate the number and value of bids they

would optimally make next round (if no other bidschanged)

b) AMs adjust their bidding towards this optimalnumber/value using Approach 1, 2 or 3 fromprevious section

4) Return to step 2)

In order to capture the different incentives behind servicedeployment decisions, we consider three different kinds ofAMs, which we denote as QoS-sensitive, Price-sensitive andBackground. Whereas the first two solve (11) to arrive tothe optimal QoS/resource cost tradeoff embodied by (8),Background AMs generate bids and bid values drawn fromstatic probability distributions. Hence, Background helps usmodel additional EZ load that is non-adaptive. Regarding QoS-sensitive and Price-sensitive AMs, the only difference is the

9

TABLE IISIMULATION PARAMETERS

N_EZ Number of execution zones 20N_App_QoS Number of QoS-sensitive apps

(ζ = 30000)10, 20

N_App_Price Number of Price-sensitive apps(ζ = 5000)

10, 20

N_App_BG Number of Background apps 10, 20N_Srv_Slots Number of service slots Mi in an exe-

cution zone50, 100

Mu_MaxMu_Min

Maximum and minimum values for therate at which a given execution zone canprocess user requests

5.53.5

Lambda_MaxLambda_Min

Maximum and minimum values for therate at which user requests for a givenapplication are generated at a given ex-ecution zone

11

N_Iterations Maximum number of epochs for a givensimulation replica

100

N_App_Threads Number of application manager threadsin the bid planning pool

450

N_EZ_Threads Number of application manager threadsin the execution pool

450

N_Replicas Number of simulation replicas per sim-ulation run

1

N_Hist_Bins Number of histogram bins to representFV i

(vij)

60

chosen value of ζ. For QoS-sensitive AMs we use ζ = 30000;for Price-sensitive AMs we use ζ = 5000. These values arearbitrary and only chosen to differentiate between these twocategories of AMs.

We parametrize our simulations in terms of the variablespresented in Table II. In order to reliably test the statisticalproperties of our simulation, we organize our simulations intoruns, with each run consisting of a set of N_Replicas simu-lation replicas. For each one of these replicas we use the samevalues for all parameters and report the averages. For eachsimulation run, we choose a combination of individual valuesof N_EZ, N_App_QoS, N_App_Price, N_App_BG andN_Srv_Slots; the rest of the parameters are kept unchangedfor all simulation runs. The values used for these variables areshown in the third column of Table II. In order to representthe probability densities FV

i

(vij), we used histograms thatsplit the entire range for vij into N_Hist_Bins equally-sizedbins. Additional information on our simulation setup can befound below.

B. Competition, QoS and Pricing

The first variable that we examine is the average costincurred by each AM. Each subgraph in Fig. 4 correspondsto a combination of values as defined in Table II (see thetop- and right-most edges of each panel). Because resourceallocation is performed by means of a Vickrey auction, thetotal cost is defined by the losing bids in each EZ (the readeris reminded that in our multi-item Vickrey auction modela bidder winning n bids will pay the sum of the first nlosing bids). Hence, in situations where there is a significantoverprovisioning of resources, cost will remain close to zero.Such is the case for N_App_QoS = 10, N_App_Price = 10and N_App_BG = 10; in this case, the cost paid is very lowfor all AM classes. As competition increases, however, we seethat Quality-sensitive managers are much more aggressive in

trying to secure EZ resources, and consequently end up payinghigher costs. Although Price-sensitive AMs are more willingto tolerate higher blocking probabilities, they still implement acost-quality tradeoff and hence experience consistently lowerblocking probabilities than Background applications (see Fig.5). It is interesting to note that the increase in total cost is notrelated only to competition with other adaptive AMs; evenan increase in Background leads to an increase in total cost.This happens because adaptive applications will respond toincreased competition to Background applications accordingto their cost-benefit tradeoff.

With regards to client blocking probability, Quality-sensitivemanagers achieve consistently better quality than Price-sensitive AMs (and, consequently, than Background services).It is interesting to note that, in the early stages of all simulationruns (lower values for the epoch), Background applicationsachieve lower blocking probabilities than either Price-sensitiveor Quality-sensitive managers. This is due to the fact that thesetwo categories of AMs progressively increase their number ofbids for each EZ and their bid values. Hence, whereas at thebeginning Background applications occupy a majority of theavailable service slots, they are soon displaced by adaptiveAMs. This is shown explicitly in Fig 6. Although Backgroundmanagers send an approximately constant number of bids toeach EZ the number of bids they actually win decreases sig-nificantly as both Price-sensitive and Quality-sensitive adaptto the increased competition. However, this adaptation doesnot consist on indiscriminately increasing the number ofsent bids; our approach allows managers to concentrate theirresources on those EZs that will bring them a greater benefit(a greater decrease in client blocking probability) at a smallerexpected resource cost. For example, in the case in whichN_App_QoS = 10, N_App_Price = 10 and N_App_BG =20, the number of bids sent by Price-sensitive and Backgroundmanagers is very similar. However, the blocking probabilityexperienced by Price-sensitive applications is much lower thanthat experienced by Background applications. There are tworeasons for this. The first one is that, as shown in Fig. 6, Price-sensitive managers are winning more bids than Backgroundmanagers; the second one, that our approach drives adaptiveAMs to bid more in those EZs that provide better cost/benefit.

C. Scalability Considerations

One of the challenges faced by many auction-based systemsis that they can exhibit high communications and/or computa-tional complexity [17]. This means that, depending on the wayin which the auction is resolved, a very large number of eithermessages or computations are required to achieve an optimalresult. Hence, many of our engineering choices will be aimedto reducing the communications and computational overheadof the system. First, as detailed above, we eschew globaloptimality in which a central optimizer performs the optimalallocation in favour of a system based on local optima for eachEZ. This breaking of the entire market into a number of localmarkets allows the computational problem to be massivelyparallelized, and has been shown to impose a tolerable impacton the quality of the solution [26]. Our use of service slots to

10

Fig. 4. Per-epoch application manager cost (arbitrary units)

model the way in which computational resources are offeredto applications is also useful to scalability. In addition todiscretizing the resource allocation problem, it also provides asimplified way in which the execution zones can advertise theiravailable capacity and hence reduces the size of the messagesrequired to that effect. Another benefit of the use of serviceslots is that it obviates the need for a bidding language todescribe the various different bundles available in a multi-item auction; in our case, only the number of service slotsrequired is needed. Finally, we constrain each bidder fromassigning a bid value to each service slot they bid for; instead,bidders specify a single price and the number of service slot forwhich they bid at that price. This means that, when bidding forresources, each AM only needs to send the number of serviceslots bidded for and the value to be used for these bids. If thenumber of bids is constrained to an 8-bit integer and the valueto bid is discretized to one of ∼ 28 levels, a single 16-bit wordis sufficient for each bid.

Another common problem with auction-based systems isthat bidders can experience unpredictable results, dependingon the number and value of bids received by the auctioneer.To help AMs reduce their uncertainty, we provide closed-formsolutions of the expected outcomes of interest to AMs that

are easy to calculate and depend on few inputs. In particular,AMs can evaluate their cost-benefit for each EZ only basedon knowledge of the CDF of bid values for each auctioneer,the number of bids received by it, and the number of serviceslots it has on offer. By conveniently discretizing these to ∼ 28

levels, the entire CDF can comfortably fit in a single IP packet.In particular, assume that each EZ reports 28 bin values andthat these have been normalized so that the count per bin canbe represented again with ∼ 28 levels. Then, a full CDF can berepresented in 211 bits with no compression, which amounts toa ∼256 byte data structure. Since each AM will receive onesuch packet from each EZ, this amounts to a traffic streamof ∼2 ×N_EZ kilobits/epoch. For a representative value ofN_EZ=1000 and and an epoch of 10 seconds, this amounts to200 kbps.

With regards to computation, a conventional laptop dual-core CPU running at 2.50GHz achieves a throughput of ∼1500simulation units per second, where each unit includes bothbid planning for a single AM and Vickrey auction resolutionfor a single EZ. Even if we disregard CPU time devoted toauction resolution, this still implies a throughput of ∼1500EZs per second using low-end hardware. Hence, a load ofN_EZ = 1000 with an epoch length of 10 seconds as suggested

11

Fig. 5. Per-epoch execution zone blocking probability

above would induce a CPU load of only ∼7% on our referencemachine. Based on these calculations, a virtual machine with 4cores running at 2.50GHz could then act as AM for a systemwith N_EZ ≈ 60,000; we posit that this makes our systemeconomical.

D. Discussion

The convergence approaches used (see Section III-F) wereall tested in simulation with a variety of parameters. Forreasons of space, only one of these can be presented in thispaper. The cases studied were Approach 2 with values of wbetween 0.05 and 0.5 and Approach 3 with similar valuesfor w but with d between 0.9 and 0.99. In these cases thefinal results (after all iterations) were nearly identical. Twoexceptions were noted. Using Approach 3 (where the rate ofchange of bids shrank with time) the algorithm could simplystop moving bids too soon and end on a very different resultif w shrank quickly (low values of d). This is as would beexpected since that algorithm is designed to stop all changesin bidding as time progresses. With Approach 2 then highervalues of w (for example 0.5) give end results which oscillateslightly more than the results shown but still giving a similarpattern overall. Higher values of w failed to converge but this is

completely expected as that algorithm allows extremely severechanges every iteration.

In all cases studied the transient results for the first tento twenty epochs differed with the parameters. The majordifference is observed in the number of sent bids. The resultsin this paper are all for Approach 2 with w = 0.1, whichshow an initial ”overshoot”. If Approach 1 is used instead,this overshoot is substituted for a slow climb; this happensbecause Approach 1 seeks the equilibrium position moreslowly. This is best illustrated by the results for 10 backgroundslots per AM, 100 slots per EZ (the second column in allresults). In these figures a transient can be seen in epochs5–20 which is not present in slower converging methods forreaching equilibrium. If w = 0.5 is used with Approach 2,this overshoot is much larger. In most cases, however, thedifference between the three approaches was for the first 10or 20 epochs only.

Our approach assumes that bidding cycles, comprised ofmultiple epochs, are undertaken in the context of stable trafficdemand such that bidders remain a fixed group, and theirinternal service slot valuations remain fixed. We refer to [27]where dynamically changing traffic patterns can be consideredconstant within 10 minute intervals. Our results show that

12

Fig. 6. Per-epoch average won bids for application managers of given type (thousands)

the bidding process converges at ∼ 20 epochs, which is wellwithin the period of traffic stability with epochs of 10s as wehave assumed in this paper.

We have made every effort to keep the number of param-eters used in the model to a minimum in order to reducethe parameter space. However, it remains a model with anumber of free parameters. Of these, our plots explore themost important: the number of applications in the ecosystem(in terms of background, price sensitive and QoS sensitive).The number of iterations per simulation and the number ofsimulation replicas are only important to ensure convergenceis reasonably likely and that the results of a run are not merelya result of random processes; they do not provide interestinginformation per-se. This is also the case for other parameterssuch as µ, λ their ratio ρ; the number of EZs; and the ratio ofζ values between different types of adaptive applications. Allof these are fixed for our simulations in the interest of spacebecause they did not provide additional information.

As would be expected the results showed that AMs fo-cussing on QoS ended up paying more than those focusingon reducing price. This difference was greater when moreAMs were competing for fewer resources. In return thoseAMs prioritising QoS obtained a lower loss probability. In

extreme cases, the background AMs which did not vary theirbehavior obtained extremely poor loss probabilities (nearing100%) when there was a great deal of competition for a smallnumber of slots. This was because they won very few bids(indeed in some scenarios their mean number of won bids wasfar less than 1 – that is in a typical epoch they won no bids).Only in the scenarios where there was very little competitiondid the QoS and the price sensitive AMs obtain similar resultsto each other. In particular when there are 10 AMs of eachtype competing for 100 EZ slots then there is little overall lossand the QoS and price sensitive AMs end up with very similarcosts and QoS.

V. RELATED WORK

Due to the useful properties that we have briefly describedelsewhere in the paper, there is a large body of work onauction-based resource allocation. General analysis of the topiccan be found in works focusing on the game theoretic aspectsof multi agent systems, such as [28] sec. 11.2.3 and [17] sec.9.3 to name but two examples. We now provide concrete andrepresentative examples of related work in the literature, andhow they relate to our current proposal.

In [29] the authors describe a double auction scenariobetween mobile users and cloudlets. Examples of these kind

13

rely on a trusted third party to administer the trading. Our ap-proach is architecturally different in that a logically centralizedmatchmaking entity is not involved in the resource allocationdecision; instead, the system is self-organized. There have alsobeen proposals for decentralized markets. An example of thisis CompuP2P [6], a system that implements an open market forpeer resources quantized into different markets. Each marketis managed by a particular peer through a dynamic hash table(DHT), and pricing is arrived at by using Vickrey auctions[16]. Our proposed architecture relies on direct communica-tions rather than on a DHT, thus simplifying the protocolsand avoiding the need for costly and time-consuming overlaymaintenance operations. Vickrey auctions have also been usedfor virtual network embedding [30], where service providerslease resources from multiple infrastructure providers forconstructing virtual topologies across autonomous systems. Inthis case, infrastructure providers bid for the virtual resourcesthey are willing to host according to the requirements ofthe service provider. A distinct difference with our work isthat bidding concerns communication resources as opposed tocompute resources, but also the fact that the physical resourceproviders are the bidders rather than the auctioneers. Anothersystem based on Vickrey auctions is Spawn [7], which usesan open market to solve a distributed CPU resource allocationproblem. Rather than true money, Spawn uses an abstractform of priority, so that better funded processes can obtaincorrespondingly better access to the computing infrastructurethan others. A limitation of Spawn is that it may not scale wellwith spatial price dynamics; the scalability of our approachunder price variability among EZs is guaranteed by the auctionresolution process and demonstrated in simulation.

Some contributions to the literature have moved away fromsimple Vickrey auctions to consider multi-item auctions. Full-blown combinatorial auctions, in which bidders express theirpreferences for product bundles, are more versatile but canbe very computationally intensive and involve large delays[17]. Hence, multiple simplifications have been proposed.In [10], each participant allocates its finite budget to bidon a given resource set, and receives a proportion of eachresource commensurate with the proportion of its bid with thebids of other participants; this same technique was later onproposed by [11] as a replacement for the unchoking policyof BitTorrent. Another auction-based resource managementimplementation is Mirage [12], where combinatorial auctionsusing a centralized virtual currency environment are used forsensor network testbed resource allocation. Our work presentsyet another take on computationally tractable multiunit auc-tions which is efficient, as evidenced by our simulation results.

A first set of server placement techniques has been devel-oped in the context of data centers. Their key rationale is tominimize data center energy consumption by maximizing theutilization of the running servers through efficient placementof the work load (in the form of virtual machines). Variousdecision parameters have been studied, such as the workloadof the virtual machines or their memory consumption [31][32] or the type of resources requested [13]. However, thesealgorithms are not directly applicable to our problem defini-tion, as in the data center case the effect of the underlying

physical network topology is negligible: inside data centers,dedicated high-bandwidth and low-latency links can be used.Another contribution in this direction is [14], where the authorspropose a centralized service placement algorithm requiringinformation on the entire network topology. The algorithmmaximizes the satisfied demand and minimizes the numberof placement changes. In [33] and [34], the authors proposea decentralized variant of this algorithm that overcomes thescalability limitations of [14] by performing a local opti-mization on each server, based on state information fromthat servers neighborhood. A decentralized solution was alsoproposed in [35], which uses a round-based gossip protocolfor exchanging state information between servers. In an effortto combine the optimality and scalability benefits of cen-tralized and distributed solutions, respectively, a hierarchicalmodel was investigated in [36] and [37]. However, the aboveapproaches do not take into account latency or any othercharacteristics of the underlying physical network. A dynamicand latency-aware service placement algorithm for assigningresources to services is presented in [38]. The servers areconnected via a peer-to-peer overlay technology. Each serveris responsible for both taking part in the management tasks andrunning a subset of the available services. A comprehensivereview of various service placement approaches can be foundin [39]. In general, resource allocation in non-auction basedapproaches is predominantly driven by the objectives of theinfrastructure provider, eg. balancing resource usage, reducingenergy consumption, etc. The absence of customer influence inthis process can compromise the expected service quality butalso allow providers to dictate the price of resources, whichmight not be fair.

The techniques discussed above require a mapping of indi-vidual workloads (services) to a set of servers. Within thefield of distributed systems, the replica placement problemhas also been studied extensively, i.e. where to cache orreplicate popular data objects [15]. A model taking QoS intoconsideration is presented in [40]. Parameters like storagecost, update cost and access cost of data replication are usedfor the replica placement. Being NP-complete, two heuristicalgorithms are presented for the model. For tree networks, twonew policies for QoS-aware placement are presented in [41].

VI. CONCLUSIONS

Cloud applications have different requirements in terms ofcost and quality, and different applications will coexist in thesame cloud infrastructure that have different tradeoffs betweenthese. The mechanism presented in this article allows differentapplications to implement their own tradeoffs transparentlyby abstracting away the task of allocating resources betweenapplications with differing requirements to an auction mech-anism. This allows the system to be responsive to the needsof each service, without the need for specialized treatment.In addition, because applications themselves use the pricesprovided by the system as signals to drive their own cost-quality tradeoffs, they experience better outcomes than thosethat would be possible in a static system.

Many properties of the system are predictable, because theyare rooted on closed-form solutions of the Vickrey auction, a

14

well-researched mechanism. In addition, we carry out exten-sive simulations to verify that the system as a whole (thatis, the global market in which applications bid for resourcesaccording to their needs) is both stable and efficient. We findthat it is stable in the sense that, if the input parameters of thesystem do not change, it converges to an equilibrium state. Itis efficient in the sense that resources are allocated to thoseapplications that need them the most, and in those EZs wherethey provide the best improvement to the quality experiencedby its users.

In the process of reaching an efficient equilibrium, thesystem will track any changes in its inputs, and hence, it willadapt to the changing needs of those applications using it.This provides self-tuning, that is, the ability of the system tonaturally configure its resources to best fit the demand profilesimposed by heterogeneous populations of cloud users.

ACKNOWLEDGMENT

This research has received funding from the Seventh Frame-work Programme (FP7/2007-2013) of the European Union,through the FUSION project and the FLAMINGO Networkof Excellence (grant agreements 318205 and 318488).

REFERENCES

[1] L. M. Vaquero and L. Rodero-Merino, “Finding your way in the fog:Towards a comprehensive definition of fog computing,” SIGCOMMComput. Commun. Rev., vol. 44, no. 5, pp. 27–32, Oct. 2014. [Online].Available: http://doi.acm.org/10.1145/2677046.2677052

[2] N. Nisan, T. Roughgarden, E. Tardos, and V. V. Vazirani, AlgorithmicGame Theory. New York, NY, USA: Cambridge University Press, 2007.

[3] K. Krauter, R. Buyya, and M. Maheswaran, “A taxonomy and surveyof grid resource management systems for distributed computing,” Softw.Pract. Exper., vol. 32, no. 2, pp. 135–164, Feb. 2002.

[4] W. Wang and B. Li, “Market-driven bandwidth allocation in selfishoverlay networks,” in Proc. of IEEE INFOCOM, 2005, pp. 2578–2589.

[5] R. Landa, R. Clegg, E. Mykoniati, D. Griffin, and M. Rio, “A SybilproofIndirect Reciprocity Mechanism for Peer-to-Peer Networks,” in Proc. ofINFOCOM, 2009.

[6] R. Gupta, V. Sekhri, and A. K. Somani, “CompuP2P: An Architecture forInternet Computing Using Peer-to-Peer Networks,” IEEE Trans. Paralleland Distrib. Syst. (TPDS), vol. 17, no. 11, 2006.

[7] C. A. Waldspurger, T. Hogg, B. A. Huberman, J. O. Kephart, and W. S.Stornetta, “Spawn: A Distributed Computational Economy.” IEEE Trans.Software Eng., vol. 18, no. 2, pp. 103–117, 1992.

[8] J. Shneidman, C. Ng, D. C. Parkes, A. AuYoung, A. C. Snoeren,A. Vahdat, and B. Chun, “Why Markets Could (But Don’t Currently)Solve Resource Allocation Problems in Systems,” in Proc. of ACMHotOS. USA: USENIX - (TCOS), June 2005.

[9] Y. Fu, J. Chase, B. Chun, S. Schwab, and A. Vahdat, “SHARP: anarchitecture for secure resource peering,” in Proc. of ACM Symp. onOp. Sys. Princ. (SOSP), 2003.

[10] M. Feldman, K. Lai, and L. Zhang, “A Price-Anticipating ResourceAllocation Mechanism for Distributed Shared Clusters,” in Proc. ACMEC, Jun. 2005.

[11] D. Levin, K. LaCurts, N. Spring, and B. Bhattacharjee, “BitTorrent isan auction: analyzing and improving BitTorrent’s incentives,” in Proc.of SIGCOMM, 2008, pp. 243–254.

[12] B. N. Chun, P. Buonadonna, A. AuYoung, C. Ng, D. C. Parkes,J. Shneidman, A. C. Snoeren, and A. Vahdat, “Mirage: A Microeco-nomic Resource Allocation System for SensorNet Testbeds,” in Proc. ofEmNetsII, 2005.

[13] M. Stillwell, D. Schanzenbach, F. Vivien, and H. Casanova, “Resourceallocation algorithms for virtualized service hosting platforms,” J. Par-allel Distrib. Comput., vol. 70, no. 9, pp. 962–974, Sep. 2010.

[14] A. Karve, T. Kimbrel, G. Pacifici, M. Spreitzer, M. Steinder, M. Sviri-denko, and A. Tantawi, “Dynamic placement for clustered web applica-tions,” in Proc. of ACM WWW, USA, 2006, pp. 595–604.

[15] S. Zaman and D. Grosu, “A Distributed Algorithm for the ReplicaPlacement Problem,” IEEE Trans. Parallel and Distrib. Syst. (TPDS),vol. 22, no. 9, pp. 1455–1468, 2011.

[16] W. Vickrey, “Counterspeculation, Auctions, and Competitive SealedTenders,” The Journal of Finance, vol. 16, no. 1, pp. 8–37, 1961.

[17] P. Cramton, Y. Shoham, and R. Steinberg, Combinatorial Auctions. MITPress, 2006.

[18] M. Yokoo, “False-name bids in combinatorial auctions,” ACM SIGecomExchanges, vol. 7, no. 1, pp. 48–51, 2007.

[19] Q. Wang, B. Ye, B. Tang, S. Guo, and S. Lu, “ebay in the clouds:False-name-proof auctions for cloud resource allocation,” in DistributedComputing Systems (ICDCS), 2015 IEEE 35th International Conferenceon. IEEE, 2015, pp. 153–162.

[20] H. A. David and H. N. Nagaraja, Order Statistics, ser. Wiley Series inProbability and Statistics. Wiley-Interscience, 2003.

[21] N. Balakrishnan and C. R. Rao, Order Statistics: Theory and Methods,ser. Handbook of Statistics. Elsevier, 1998, vol. 16.

[22] D. Bertsekas and R. Gallager, Data Networks, 2nd ed. P. Hall, 1992.[23] D. Bertsekas, “Dynamic behavior of shortest path routing algorithms for

communication networks,” IEEE Trans. on Auto. Control, vol. 27, no. 1,pp. 60–74, 1982.

[24] Z. Wang and J. Crowcroft, “Analysis of Shortest-path Routing Al-gorithms in a Dynamic Network Environment,” SIGCOMM Comput.Commun. Rev., vol. 22, no. 2, pp. 63–71, Apr. 1992.

[25] R. P. Garg and I. Sharapov, Techniques for Optimizing Applications -High Performance Computing. Prentice-Hall, 2002.

[26] S. M. Kakade, M. Kearns, and L. E. Ortiz, “Graphical Economics,”Learning Theory (Springer LNCS), vol. 3120, pp. 17–32, 2004.

[27] P. Wendell, J. W. Jiang, M. J. Freedman, and J. Rexford., “DONAR:Decentralized Server Selection for Cloud Services,” in SIGCOMM,2010.

[28] Y. Shoham and K. Leyton-Brown, Multiagent Systems: Algorithmic,Game-Theoretic, and Logical Foundations. Cambridge UniversityPress, 2009.

[29] A. Jin, W. Song, P. Wang, D. Niyato, and P. Ju, “Auction MechanismsToward Efficient Resource Sharing for Cloudlets in Mobile CloudComputing,” Transactions on Services Computing, 2015.

[30] F.-E. Zaheer, J. Xiao, and R. Boutaba, “Multi-provider service negotia-tion and contracting in network virtualization,” in Network Operationsand Management Symposium (NOMS), 2010 IEEE, 2010, pp. 471–478.

[31] J.-W. Jang, M. Jeon, H.-S. Kim, H. Jo, J.-S. Kim, and S. Maeng,“Energy Reduction in Consolidated Servers through Memory-AwareVirtual Machine Scheduling,” IEEE Trans. on Computers, vol. 60, no. 4,pp. 552–564, 2011.

[32] M. B. Reynolds, D. R. Hulce, K. M. Hopkinson, M. E. Oxley, andB. E. Mullins, “A Bin Packing Heuristic for On-Line Service Placementand Performance Control,” IEEE Transactions on Network and ServiceManagement, vol. 10, no. 3, pp. 326–339, 2013.

[33] C. Adam, R. Stadler, C. Tang, M. Steinder, and M. Spreitzer, “A ServiceMiddleware that Scales in System Size and Applications,” in Proc. ofIFIP/IEEE IM (Intl. Symp. on Int. Net. Man.), 2007, pp. 70–79.

[34] C. Adam and R. Stadler, “Service middleware for self-managing large-scale systems,” IEEE Transactions on Network and Service Manage-ment, vol. 4, no. 3, pp. 50–64, 2007.

[35] F. Wuhib, R. Stadler, and M. Spreitzer, “A gossip protocol for dynamicresource management in large cloud environments,” IEEE Transactionson Network and Service Management, vol. 9, no. 2, pp. 213–225, 2012.

[36] H. Moens, J. Famaey, S. Latre, B. Dhoedt, and F. De Turck, “Designand evaluation of a hierarchical application placement algorithm in largescale clouds,” in Proc. of IFIP/IEEE IM (Intl. Symp. on Int. Net. Man.),2011, pp. 137–144.

[37] H. Moens and F. De Turck, “A scalable approach for structuring large-scale hierarchical cloud management systems,” in Proc. of IEEE/IFIPCNSM (Int. Conf. on Net. and Serv. Man.), 2013, pp. 1–8.

[38] J. Famaey, W. De Cock, T. Wauters, F. De Turck, B. Dhoedt, andP. Demeester, “A latency-aware algorithm for dynamic service placementin large-scale overlays,” in Proc. of IFIP/IEEE IM (Intl. Symp. on Int.Net. Man.), 2009, pp. 414–421.

[39] B. Jennings and R. Stadler, “Resource management in clouds: Surveyand research challenges,” Journal of Network and Systems Management,pp. 1–53, 2014.

[40] C.-W. Cheng, J.-J. Wu, and P. Liu, “QoS-aware, access-efficient, andstorage-efficient replica placement in grid environments,” J. Supercom-put., vol. 49, no. 1, pp. 42–63, Jul. 2009.

[41] A. Benoit, V. Rehn-Sonigo, and Y. Robert, “Replica Placement andAccess Policies in Tree Networks,” IEEE Trans. Parallel and Distrib.Syst. (TPDS), vol. 19, no. 12, pp. 1614–1627, 2008.

15

Raul Landa is currently a Data Scientist in theNetwork Planning and Operations group within SkyUK Network Services. His research interests involvethe measurement-based modelling of Internet videoquality of experience, the large-scale behaviour ofcontent delivery networks and the strategic aspectsin peer-to-peer overlays. He received a PhD in Elec-tronic and Electrical Engineering from UniversityCollege London, UK.

Marinos Charalambides is a senior researcher atUniversity College London. He holds a BEng, aMSc and a Ph.D. from the University of Surrey, UK.He has been working in a number of European andUK national projects since 2005 and his researchinterests include network programmability, contentdistribution, and adaptive resource management. Hehas been the technical program chair of severalevents and serves in the committees of the mainnetwork and service management conferences.

Richard G. Clegg is a Research Fellow at theDepartment of Computing in Imperial College Lon-don. His PhD in mathematics and statistics from theUniversity of York was gained in 2005. His researchinterests include investigations of the dynamic be-haviour of networks and measurement of networktraffic statistics.

David Griffin is a Principal Research Associateat University College London. He has a PhD inElectronic and Electrical Engineering from UCL.His research interests are in planning, managementand dynamic control for providing QoS in multi-service networks, p2p networking and novel routingparadigms in the Internet.

Miguel Rio is a Reader (Associate Professor) inComputer Networks at the Department of Electronicand Electrical Engineering at University CollegeLondon. He has published extensively in top rankedConferences and journals in the areas of Networkmeasurement, congestion control, new network ar-chitectures and, more recently, in the interactionbetween cloud and network services. He holds aPhD from the University of Kent at Canterbury andMSc and MEng from the Department of Informatics,University of Minho, Portugal.