KTH TECHNICAL REPORT TRITA-EE, 1653-5146; 2012:022; URN:NBN:SE:KTH:DIVA-93735; SUBMITTED FOR PUBLICATION.

Predicting response timesfor the Spotify backend

27 April, 2012

Rerngvit Yanggratoke∗, Gunnar Kreitz†, Mikael Goldmann†, and Rolf Stadler∗∗ACCESS Linnaeus Center, KTH Royal Institute of Technology, Sweden

†Spotify, SwedenEmail: {rerngvit,stadler}@kth.se{gkreitz,migo}@spotify.com

Abstract—We model and evaluate the performance ofa distributed key-value storage system that is part ofthe Spotify backend. Spotify is an on-demand musicstreaming service, offering low-latency access to a libraryof over 16 million tracks and serving over 10 millionusers currently. We first present a simplified model of theSpotify storage architecture, in order to make its analysisfeasible. We then introduce an analytical model for thedistribution of the response time, a key metric in theSpotify service. We parameterize and validate the modelusing measurements from two different testbed configu-rations and from the operational Spotify infrastructure.We find that the model is accurate—measurements arewithin 11% of predictions—within the range of normalload patterns. We apply the model to what-if scenariosthat are essential to capacity planning and robustnessengineering. The main difference between our work andrelated research in storage system performance is thatour model provides distributions of key system metrics,while related research generally gives only expectations,which is not sufficient in our case.

Index Terms—Key-value store, distributed object store,performance modeling, system dimensioning, perfor-mance measurements, response times, streaming mediaservices

I. INTRODUCTION

The Spotify service is a peer-assisted system, mean-ing it has a peer-to-peer component to offload backendservers, which are located at three sites (Stockholm,Sweden, London, UK, and Ashburn, VA). While theSpotify backend servers run a number of services,such as music search, playlist management, and socialfunctions, its core service is audio streaming, which isprovided by the Spotify storage system [1]. When aclient plays a music track, its data is obtained from acombination of three sources: the clients local cache (ifthe same track has been played recently), other Spotifyclients through peer-to-peer technology, or the Spotifystorage system in a backend site [1].

Low latency is key to the Spotify service. Whena user presses “play”, the selected track should start“instantly.” To achieve this, the client generally fetchesthe first part of a track from the backend and startsplaying as soon as it has sufficient data so that bufferunderrun (“stutter”) will be unlikely to occur. There-fore, the main metric of the Spotify storage systemis the fraction of requests that can be served withlatency at most t for some small value of t, typicallyaround 50 ms. (We sometime use the term responsetime instead of latency in this paper.)

The Spotify storage system has the functionality ofa (distributed) key-value store. It serves a stream ofrequests from clients, whereby a request provides akey and the system returns an object (e.g., a part ofan audio track). In this paper, we present an analyticalmodel of the Spotify storage architecture that allows usto estimate the distribution of the response time of thestorage system, as a function of the load, the storagesystem configuration and model parameters that wemeasure on storage servers. The model centers arounda simple queuing system that captures the criticalsystem resource (i.e., the bottleneck), namely, access tothe server’s memory cache and disk where the objectsare stored.

We validate the model (1) for two different storagesystem configurations on our laboratory testbed, whichwe load using anonymized Spotify traces, and (2) forthe operational Spotify storage system, whereby weutilize load and latency metrics from storage servers ofthe Stockholm site. We find that the model predictionsare within 11% of the measurements, for all systemconfigurations and load patterns within the confidencerange of the model. As a consequence, we can predicthow the response time distribution would change inthe Stockholm site, if the number of available serverswould change, or how the site in a different configura-tion would handle a given load. Overall, we find that asurprisingly simple model can capture the performanceof a system of some complexity. We explain thisresult with two facts: (1) the storage systems wemodel are dimensioned with the goal that access tomemory/storage is the only potential bottleneck, whileCPUs and the network are lightly loaded; (2) werestrict the applicability of the model to systems withsmall queues —they contain at most one element onaverage. In other words: our model is accurate for alightly loaded storage system. Interestingly, the modelcaptures well the normal operating range of the Spotifystorage system. As our measurements show, increasingthe load beyond the confidence limit of our analysiscan lead to a significant increase in response times fora large fraction of requests.

The paper is organized as follows. Section II brieflydescribes the Spotify storage system. Section III con-tains the analytical model of the system that suitsour purpose. Section IV describes our work on theevaluation of the model and the estimation of the

1

Fig. 1. Spotify Storage Architecture

model parameters, both on the lab testbed and on theSpotify operational system. Section V gives examplesof applying the model. Section VI discusses relatedwork and Section VII contains our conclusions andfuture work.

II. THE SPOTIFY BACKEND STORAGEARCHITECTURE

We give a (simplified) overview of the part of theSpotify backend responsible for music delivery to theclients. Its architecture is captured in Figure 1. EachSpotify backend site has the same layout of storageservers, but the number of servers varies. The masterstorage component is shared between the sites. Whena user logs in, the client connects to an Access Point

(AP) using a proprietary protocol. Through the AP,the client can access the backend services includingstorage. The client maintains a long-lived TCP con-nection to the AP, and requests to backend servicesare multiplexed over this connection.

Spotify’s storage is two-tiered. A client request foran object goes to Production Storage, a collectionof servers that can serve most requests. The protocolbetween the AP and Production Storage is HTTP, andin fact, the Production Storage servers run softwarebased on the caching Nginx HTTP proxy [2]. Theobjects are distributed over the production servicemachines using consistent hashing of their respectivekeys [3]. Each object is replicated on three differentservers, one of which is identified as the primaryserver for the object. APs route a request for an objectto its primary server. If the primary server does notstore the requested object, it will request it from oneof the replicas. If they do not store it, the requestwill be forwarded over the Internet to Master Storage

(which is based upon a third-party storage service)and the retrieved object will subsequently be cachedin Production Storage. When the primary server ofan object fails, an AP routes a request to one of thereplicas instead.

While the presentation here centers on music de-livery, we remark that the storage system deliversadditional data to Spotify clients, in particular images

Fig. 2. Simplified architecture as a basis for the performance model

(e.g., cover art for albums) and advertisements. Wealso point out that the number of requests that a clientmakes to the backend storage for each track playedvaries significantly, due to local caching and peer-to-peer functionality. As a consequence, the request ratespresented in this paper do not correspond to the actualnumber of tracks played in the Spotify system.

III. AN ANALYTICAL MODEL OF THE STORAGEARCHITECTURE

In order to make a performance analysis feasible,we develop a simplified model of the Spotify storagearchitecture (Figure 1) for a single Spotify backendsite, the result of which is shown in Figure 2. First,we omit Master Storage in the simplified model andthus assume that all objects are stored in ProductionStorage servers, since more more than 99% of therequests to the Spotify storage system are served fromthe Production Storage servers. Second, we modelthe functionality of all APs of a site as a singlecomponent. We assume that the AP selects a storageserver uniformly at random to forward an incomingrequest, which approximates the statistical behaviorof the system under Spotify object allocation androuting policies. Further, we neglect network delaysbetween APs and storage servers, because they aresmall compared to the response times at the storageservers. In the following, we analyze the performanceof the model in Figure 2 under steady-state conditionsand Poisson arrivals of requests.

A. Modeling a single storage server

In the context of a storage system, the criticalresources of a storage server are memory and diskaccess, which is captured in Figure 3. When a requestarrives at a server, it is served from memory with prob-ability q and from one of the disks with probability1− q. Assuming that the server has nd identical disks,the request is served from a specific disk with prob-ability 1/nd. We further assume that requests arriveat the server following a Poisson process with rate λ.(All rates in this paper are measured in requests/sec.)We model access to memory or a disk as an M/M/1queue. We denote the service rate of the memory byµm and that of the disk by µd, whereby µm � µd

holds.

2

Fig. 3. Queuing model of critical resources on a server

Based on Figure 3, we compute the latency T fora request on a server, which we also refer to asresponse time. (In queuing systems, the term sojourn

time is generally used for T .) The probability that arequest is served below a specific time t is given byPr(T ≤ t) = qPr(Tm ≤ t) + (1 − q)Pr(Td ≤ t),whereby Tm and Td are random variables representingthe latency of the request being served from memoryor a disk, respectively. For an M/M/1 queue in steadystate with arrival rate λ, service rate µ, and latency T ,the formula Pr(T ≤ t) = 1 − e−µ(1−λ/µ)t holds [4].Therefore, we can write

Pr(T ≤ t) = q(1− e−µm(1−λm/µm)t)

+(1− q)(1− e−µd(1−λd/µd)t),whereby λm is the arrival rate to the memory queueand λd to a disk queue. Typical values in our exper-iments are t ≥ 10−3 sec, λm ≤ 103 requests/sec,and µm ≥ 105 requests/sec. We therefore approximatee−µm(1−λm/µm)t with 0. Further, since λm = qλ andλd = (1− q)λ/nd hold, the probability that a requestis served under a particular latency t is given byPr(T ≤ t) = q + (1− q)(1− e−µd(1−(1−q)λ/µdnd)t).

(1)

B. Modeling a storage cluster

We model a storage cluster as an AP and a set Sof storage servers, as shown in Figure 2. The loadto the cluster is modeled as a Poisson process withrate λc. When a request arrives at the cluster, it isforwarded uniformly at random to one of the storageservers. Let Tc be a random variable representingthe latency of a request for the cluster. We get thatPr(Tc ≤ t) =

�s∈S

1|S|Pr(Ts ≤ t), whereby Ts is a

random variable representing the latency of a requestfor a storage server s ∈ S.

For a particular server s ∈ S, let µd,s be the servicerate of a disk, nd,s the number of identical disks,and qs the probability that the request is served frommemory. Let f(t, nd, µd,λ, q) := Pr(T ≤ t) definedin equation 1. Then, the probability that a request tothe cluster is served under a particular latency t isgiven by

Pr(Tc ≤ t) =1

|S|�

s∈S

f(t, nd,s, µd,s,λc

|S| , qs). (2)

The above model does not explicitly account forthe replications of objects. As explained in Section II,the Spotify storage system replicates each object three

times. The AP routes a request to the primary serverof the object. A different server is only contacted, ifthe primary server either fails or does not store theobject. Since both probabilities are small, we considerin our model only the primary server for an object.

IV. EVALUATION OF THE MODEL ON THE LABTESTBED AND THE SPOTIFY OPERATIONAL

ENVIRONMENT

A. Evaluation on the lab testbed

The first part of the evaluation is performed on ourthe KTH lab testbed, which comprises some 60 rack-based servers interconnected by Ethernet switches. Weuse two types of servers, which we also refer to assmall servers and large servers (Table I).

1) Single storage server: In this series of exper-iments, we measure the response times of a singleserver as a function of the request rate, and we estimatethe model parameters. This allows us to validate ouranalytical model of a single server, as expressed inequation 1. The requests are generated using a Spotifyrequest trace, and they follow a Poisson arrival process.A request retrieves an object, which has an averagesize of about 80 KB.

The setup consists of two physical servers, a loadinjector and a storage server, as shown in Figure4 (top). Both servers have identical hardware; theyare either small or large servers. All servers runUbuntu 10.04 LTS. The load injector has installed acustomized version of HTTPerf [5]. It takes as inputthe Spotify trace, generates a stream of HTTP requests,and measures the response times. The storage serverruns Nginx [2], an open-source HTTP server.

Before conducting an experiment, we populate thedisk of the storage server with objects. A small serveris populated with about 280K objects (using 22GB), alarge one with about 4,000K objects (using 350GB).Each run of an experiment includes a warm-up phase,followed by a measurement phase. During warm up,the memory is populated with the objective to achievethe same cache hit ratio as the server would achieve insteady state, i.e., after a long period of operation. Thememory of the small server holds some 12K objects,the memory of the large server some 750K objects.We use a Spotify request trace for the warmup, with30K requests for the small server and 1,000K requestsfor the large server. (The number of requests is higherfor the large server, because it has more memory.)The measurement runs are performed with a differentSpotify trace than the warmup runs. A measurementrun includes 30K requests for the small server and100K requests for the large server.

We perform a series of runs. The runs start at arequest rate of 10 and end at 70, with increments of 5,for the small server; they start at a rate of 60 and endat 140, with increments of 20, for the large server. Theresponse time for a request is measured as indicatedin Figure 4 (bottom). Figures 6a and 6b show themeasurement results for three selected latencies for the

3

Fig. 4. Setup and measurement of request latency for a singleserver.

Small server SpecificationModel Dell Power edge 750 1U serverRAM 1GBCPU Intel(R) Pentium(R) 4 CPU 2.80GHzHarddisk Single disk 40GBNetwork Controller Intel 82547GI Gigabit Ethernet Con-

troller

Large server SpecificationModel Dell PowerEdge R715 2U Rack ServerRAM 64GBCPU two 12-core AMD Opteron(tm) proces-

sorsHarddisk Single disk - 500GBNetwork Controller Broadcom 5709C Gigabit NICs

TABLE ISERVERS ON THE KTH TESTBED

small and large server, respectively. The vertical axisgives the fraction of requests that have been servedwithin a particular latency. The horizontal axis givesthe rate of the request arrival process. All measurementpoints in a figure that correspond to the same requestrate result from a single run of an experiment.

The figures further include the model predictionsin forms of solid lines. These predictions come fromthe evaluation of equation 1 and an estimation ofthe model parameters/confidence limits, which will bediscussed in Section IV-C.

We make three observations regarding the measure-ment results. First, the fraction of requests that canbe served under a given time decreases as the loadincreases. For small delays, the relationship is almostlinear. Second, the model predictions agree well withthe measurements below the confidence limit. In fact,measurements and models diverge at most 11%. Athird observation can not be made from the figures butfrom the measurement data. As expected, the varianceof the response times is small for low requst rates andbecomes larger with increasing rate. For instance, forthe small server, we did not measure any responsetimes above 200 msec under a rate of 30; however,at the rate of 70, we measured several response timesabove 1 sec.

2) A cluster of storage servers: In this series ofexperiments, we measure the response times of a

Fig. 5. Setup and measurement of request latency for clusters.

cluster of storage servers as a function of the requestrate, and we estimate the model parameters. Thisallows us to validate our analytical model of a clusteras expressed in equation 2. Similar to the single-serverexperiments, the requests are generated using a Spotifyrequest trace, and they follow a Poisson arrival process.

We perform experiments for two clusters: one clus-ter with small servers (one load injector, one AP,and five storage servers) and one with large servers(one load injector, one AP, and three storage servers).The testbed setup can be seen in Figure 5 (top). Thesoftware setup of the load injector and storage servershave been discussed above. The AP runs a customizedversion of HAProxy [6] that forwards a request to astorage server and returns the response to the loadinjector.

Before conducting an experiment, we populate thedisks of the storage servers with objects: we allocateeach object uniformly at random to one of the servers.We then create an allocation table for request routingthat is placed in the AP. This setup leads to a systemwhose statistical behavior closely approximates that ofthe Spotify storage system. During the warmup phasefor each run, we populate the memory of all storageservers in the same way as discussed above for a singleserver. After the warmup phase, the measurement runis performed. Driven by a Spotify trace, the loadinjector sends a request stream to the AP. Receivinga request from the load injector, the AP forwards itto a storage server according to the allocation table.The storage server processes the request and sends aresponse to the AP, which forwards it to the load injec-tor. The response time of each request is measured asshown in Figure 5 (bottom). Regarding dimensioning,the number of allocated objects per server is similar tothe one in the experiments discussed above involvingsingle servers. The same is true regarding the numberof objects cached in memory, the number of requests

4

(a) Single small server (b) Single large server

(c) Cluster of small servers (d) Cluster of large serversFig. 6. Lab testbed measurements and model predictions

for a warmup run, and the number of requests for ameasurement run.

For each experimental run, a request stream isgenerated at a certain rate, and, for each request,the response time is measured. The runs start at arequest rate of 50 and end at 300, with incrementsof 50, for the cluster of small servers; they start ata rate of 180 and end at 360, with increments of60, for the cluster of large servers. Figures 6c and6d show the measurement results for three selectedlatencies for the cluster of small and large servers,respectively. The figures further include the modelpredictions in form of solid lines. The predictionsare obtained from equation 2 and model parameters,discussed in Section IV-C. Our conclusions from theexperiments on the two clusters are similar to thoseon the single servers: the fraction of requests that canbe served under a given time decreases as the loadincreases. The relationship is almost linear; the slopesof the curves decrease slightly with increasing requestrate. Further, the measurements and models diverge atmost 9.3% below the confidence limit.

B. Evaluation on the Spotify operational environment

For this evaluation, we had access to hardware anddirect, anonymized measurements from the Spotifyoperational environment. The single server evaluationhas been performed on a Spotify storage server, andthe cluster evaluation has been performed with mea-surement data from the Stockholm backend site.

1) Single storage server: We benchmark an op-erational Spotify server with the same method asdiscussed in Section IV-A1. Such a server stores about7.5M objects (using 600GB), and a cache after thewarm-up phase contains about 375K objects (using30GB). (The actual capacity of the Spotify server issignificantly larger. We only populate 600GB of space,

since the traces for our experiment contains requestsfor objects with a total size of 600GB.) For a run ofthe experiment, 1000K requests are processed duringthe warm-up phase, and 300K requests during themeasurement phase. The runs start at a request rate of100 and end at 1,100, with increments of 100. Figure7a shows the measurement results for three selectedlatencies for the Spotify operational server.

The qualitative observations we made for the twoservers on the KTH testbed (Section IV-A1) holdalso for the measurements from the Spotify server.Specifically, the measurements and model predictionsdiverge at most 8.45%, for request rates lower than themodel confidence limit.

2) Spotify storage system: For the evaluation, weuse 24 hours of anonymized monitoring data from theStockholm site. This site has 31 operational storageservers. The monitoring data includes, for each storageserver, measurements of the arrival rate and responsetime distribution for the requests that have been sentby the APs. The measurement values are five-minutesaverages. The data includes also measurements fromrequests that have been forwarded to the Master Stor-age, but as stated in Section III, such requests are rare,below 1% of all requests sent to the storage servers.

Some of the servers at the Stockholm site have aslightly different configuration from the one discussedabove. These differences have been taken in accountfor the estimation of model parameters. Figure 7bpresents the measurement results in the same form asthose we obtained from the KTH testbed. It allows usto compare the performance of the storage system withpredictions from the analytical model. Specifically,it shows measurement results and model predictionsfor three selected latencies, starting at a request rateof 1,000 and ending at 12,000, with increments of1,000. The confidence limit is outside the measurement

5

Parameter Small server Large server Spotify serverµd 93 120 150nd 1 1 6α 0.0137 0.00580 0.000501q0 0.946 1.15 0.815

TABLE IIMODEL PARAMETERS FOR A SINGLE STORAGE SERVER

interval, which means that we have confidence thatour analytical model is applicable within the completerange of available measurements.

We make two observations. First, similar to theevaluations we performed on the KTH testbed, themeasurements and the model predictions diverge atmost 9.61%. This is somewhat surprising, since thisoperational environment is much more complex andless controllable for us than the lab testbed. For in-stance, for our testbed measurements, (1) we generaterequests with Poisson arrival characteristics, whichonly approximates arrivals in the operational system;(2) on the testbed we use identical servers, whilethe production system has some variations in theserver configuration; (3) the testbed configurations donot consider Master Storage, etc. Furthermore, themeasurements suggested that the fraction of requestsunder a specific latency stays almost constant withinthe range of request rates measured. In fact, our modelpredicts that, the fraction of requests served within 50msec stays almost constant until the confidence limit,at about 22,000 requests/sec. Therefore, we expect thatthis site can handle a much higher load than observedduring our measurement period, without experiencinga significant decrease in performance when consider-ing the 50 msec response-time limit.

C. Estimating model parameters / confidence limit

We determine the model parameters for the singleserver, given in equation 1, namely, the service rateof a disk µd, the number of identical disks nd, andthe probability that a request is served from mem-ory q. While nd can be read out from the systemconfiguration, the other two parameters are obtainedthrough benchmarking. We first estimate the averageservice time Ts of the single disk through runningiostat [7] while the server is in operation (i.e. afterthe warm-up phase), and we obtain µd = 1/Ts. Weestimate parameter q as a fraction of requests thathave a latency below 1 msec while the server is inoperation. Figure 8 shows the measured values for q,for different server types and request rates. Based onthe results in Figure 8 and other measurements, weapproximate, through least-square regression, q withthe linear function q = −αλ + q0, whereby λ is therequest rate. All model parameters of a single storageserver are summarized in Table II.

We now compute the model confidence limit for the

single server, i.e., the maximum request rate belowwhich we feel confident that our model (i.e., equation1) applies. Through extensive testing, we found thatour model predictions are close to the measurementsfrom the real system, as long as the average length

of any disk queue is at most one. From the behaviorof an M/M/1 queue, we know that the average queuelength for one of the disks is Ld = λd

µd−λd. Applying

the linear approximation for q and setting Ld =1, simple manipulations give the model confidencelimit λL for a single server as the positive root ofαλ2

L + (1 − q0)λL − 12µdnd = 0. The confidence

limits in Figures 6a, 6b, and 7a are computed using thismethod. As can be observed, increasing request ratesbeyond the confidence limits coincides with a growinggap between model predictions and measurements,specifically for the latency of 50 msec, which is animportant value for the Spotify storage system.

The model parameters for a cluster, appearing inequation 2, contain the model parameters of eachserver in the cluster. Therefore, if the model parametersfor each server are known, then the parameters for thecluster can be obtained.

We now discuss the model confidence limit for

the cluster, i.e., the maximum request rate to thecluster below which we have confidence that the modelpredictions are close to actual measurements, under theassumption that we know the confidence limit for eachserver. The allocation of objects to primary serversin the Spotify storage system can be approximatedby a process whereby each object is placed on aserver uniformly at random, weighted by the storagecapacity of the server. Therefore, the number of objectsallocated to servers can vary, even for a cluster withhomogeneous servers. The distribution of the numberof objects on servers can be modeled using the balls-and-bins model [8]. If the server contains a largenumber of objects, as in our system, the expected loadon the server is proportional to the number of objects.To compute the confidence limit for the cluster, wemust know the load of the highest loaded server. Aresult from the analysis of the balls-and-bins modelstates that, when m balls are thrown independently anduniformly at random into n bins and m � n · (log n)3can be assumed, then there is no bin having morethan M = m/n+

�2m logn

n (1− 1β

log logn2 logn ) balls with

high probability, for any β > 1 [9]. We apply thisresult by interpreting balls as request rates and binsas servers. By doing so, we obtain the confidencelimit λL,c of the cluster as a function of the minimumλL of the confidence limits of all servers and thenumber of servers |S|. We can conclude that theconfidence limit for the cluster is the smaller rootof 1

|S|2λ2L,c + ( 2λL

|S| − 2 log |S|Kβ,|S||S| )λL,c + λ2

L = 0,

whereby β = 2 and Kβ,|S| = 1 − 1β

log log |S|2 log |S| . The

confidence limits in Figures 6c and 6d are computedusing this method. Similar to the case of the singleserver, the model predictions can diverge significantlyfrom the measurements for rates beyond the confidencelimits.

V. APPLICATIONS OF THE MODEL

We apply the analytical model to predict, for theSpotify storage system at the Stockholm site, the frac-

6

(a) Single Spotify Storage server (b) Cluster of Spotify Storage serversFig. 7. Spotify operational environment measurements and model predictions

Fig. 8. Estimating the parameter q

tion of requests served under given latencies for a loadof 12,000 requests/sec, which is the peak load from thedataset we used. While our evaluation has involved 31servers, we use the model to estimate response timefor configurations from 12 to 52 storage servers. Theresult is shown in Figure 9a. The confidence limit is 17servers. Above this number, we have confidence thatthe model applies. We observe that for each latencycurve in the figure the slope decreases with increasingnumber of servers. This means that adding additionalservers to the storage system of, say, 20 servers resultin a much larger reduction of response time thanadding servers to the storage system of, say, 50 servers.

Second, we predict the fraction of requests servedunder specific response times for a storage system with25 servers. We consider a scenario where the loadvaries from 1,000 to 20,000 requests/sec. The resultis shown in Figure 9b. The confidence limit is 18,000below which our model applies. We observe that theslope of all curves in the figure is almost zero between1,000 to 3,000 requests/sec, beyond which it startsdecreasing. We can predict that increasing the load onthe storage system from 1,000 to 3,000 requests/secdoes not have any measurable impact on performance,while we expect that an increase from 1,000 to 15,000requests/sec clearly will. Our model also predicts that,for a response time limit of 50 msec, the fraction ofrequests remains almost unchanged for rates between1,000 and 18,000 requests/sec.

VI. RELATED WORK

Substantial research has been undertaken in model-ing the performance of storage devices (see, e.g., [10]–[12]). Our work differs from these, since our modelingwork is on the systems level and thus captures aspectsof an entire system. Several works present performancemodels of storage systems [13]–[15]. However, to the

best of our knowledge, none of them discusses andvalidates models for predicting the latency distribution

of requests to a real storage system. The authors in [13]present a performance model for Ursa Minor [16], arobust distributed storage system. Their model allowsthem to predict the average latency of a request, as wellas the capacity of the system. A second performancemodel of a storage system is presented in [14]. Inthis work, expected latency and throughput metricscan be predicted for different allocation schemes ofvirtual disks to physical storage devices. The authorsof [15] discuss in that paper a performance model forpredicting the average response time of an IO requestwhen multiple virtual machines are consolidated on asingle server.

The development and evaluation of distributed key-value stores has been an active research area. Whilethese systems generally provide more functionalitythan the Spotify storage system, to our knowledge, noperformance model has yet been developed for any ofthem. In contrast to Spotify’s storage system design,which is hierarchical, many advanced key-value stor-age systems in operation today are based on a peer-to-peer architecture. Among them are Amazon Dynamo[17], Cassandra [18], and Scalaris [19]. Facebook’sHaystack storage system follows a different designwhich is closer to Spotify’s. Most of these systemsuse some forms of consistent hashing to allocateobjects to servers. The differences in the designs of thesystems are motivated by their respective operationalrequirements, and they relate to the number of objectsto be hosted, the size of the objects, the rate of update,the number of clients, and the expected scaling of theload.

VII. DISCUSSION

We make the following contributions with this pa-per. First, we introduce an architectural model of a dis-tributed key-value store that simplifies the architectureand functionality of the Spotify storage system. Wethen present a queuing model that allows us to computethe response time distribution of the storage system.Further, we estimate the confidence range for thismodel, under the assumption that the system is lightlyloaded. Second, we perform an extensive evaluation ofthe model, first on our testbed and later on the Spotifyoperational infrastructure. This evaluation shows that

7

(a) Varying the number of servers in the storage system for a loadof 12,000 requests/sec

(b) Varying the load in the storage system for 25 storage servers

Fig. 9. Applications of the model to system dimensioning

the model predictions are accurate, with the error ofat most 11%. As it turns out, the confidence rangeof our model covers the entire operational range ofthe load to the Spotify storage system. As we havevalidated through experimentation, the performance ofthe system deteriorates when the load significantlysurpass the model-predicted confidence limit. Lastly,applying our model, we predict for a specific Spotifybackend site that the system could handle the peakload observed during a specific day with fewer servers,or, alternatively, that the system with 25 servers couldhandle a significantly higher load than observed, with-out noticable performance degradation (for importantresponse time limit, which is 50 msec).

This work is important to Spotify, since latencyis the key performance metric of its storage system.The main reason for this is that estimating latencydistributions is essential to guarantee the quality of theoverall service. Note that recent performance studieson storage systems cover only expected latencies,which is not sufficient for our case. The validity of theresults in this paper go beyond the scope of Spotify’stechnology. In fact, they can be applied to any typeof distributed key-value store that can be described bythe architectural model in this paper.

As for future work, we plan to develop a subsystemthat continuously estimates the model parameters atruntime, taking into account that the resources ofthe storage servers may be used by other processesthan object retrieval, for instance, for maintainence orsystem reconfiguration. Based on such a capability, weenvision an online performance management systemfor a distributed key-value store like the Spotify stor-age system.

ACKNOWLEDGEMENTS

Viktoria Fodor provided helpful comments and sug-gestions regarding the queuing model used in thiswork.

REFERENCES

[1] G. Kreitz and F. Niemela, “Spotify – large scale, low latency,P2P music-on-demand streaming,” in Peer-to-Peer Computing.IEEE, 2010, pp. 1–10.

[2] I. Sysoev, “Nginx,” http://nginx.org/.[3] D. R. Karger, E. Lehman, F. T. Leighton, R. Panigrahy, M. S.

Levine, and D. Lewin, “Consistent hashing and random trees:Distributed caching protocols for relieving hot spots on theworld wide web,” in STOC, 1997, pp. 654–663.

[4] L. Kleinrock, Theory, Volume 1, Queueing Systems. Wiley-Interscience, 1975.

[5] D. Mosberger and T. Jin, “httperf - a tool for measuringweb server performance,” SIGMETRICS Perform. Eval. Rev.,vol. 26, no. 3, pp. 31–37, Dec. 1998.

[6] W. Tarreau, “Haproxy,” http://haproxy.1wt.eu/.[7] S. Godard, “iostat,” http://linux.die.net/man/1/iostat.[8] M. Mitzenmacher and E. Upfal, Probability and Computing:

Randomized Algorithms and Probabilistic Analysis. Cam-bridge University Press, Jan. 2005.

[9] M. Raab and A. Steger, ““Balls into Bins” - A Simple andTight Analysis,” in Proceedings of the Second International

Workshop on Randomization and Approximation Techniques

in Computer Science, ser. RANDOM ’98, 1998, pp. 159–170.[10] J. Garcia, L. Prada, J. Fernandez, A. Nunez, and J. Carretero,

“Using black-box modeling techniques for modern disk drivesservice time simulation,” in Simulation Symposium, 2008.

ANSS 2008. 41st Annual, april 2008, pp. 139 –145.[11] A. Lebrecht, N. Dingle, and W. Knottenbelt, “A performance

model of zoned disk drives with i/o request reordering,” inQuantitative Evaluation of Systems, 2009. QEST ’09. Sixth

International Conference on the, sept. 2009, pp. 97 –106.[12] F. Cady, Y. Zhuang, and M. Harchol-Balter, “A Stochas-

tic Analysis of Hard Disk Drives,” International Journal of

Stochastic Analysis, vol. 2011, pp. 1–21, 2011.[13] E. Thereska, M. Abd-El-Malek, J. Wylie, D. Narayanan, and

G. Ganger, “Informed data distribution selection in a self-predicting storage system,” in Autonomic Computing, 2006.

ICAC ’06. IEEE International Conference on, june 2006, pp.187 – 198.

[14] A. Gulati, G. Shanmuganathan, I. Ahmad, C. Waldspurger,and M. Uysal, “Pesto: online storage performance managementin virtualized datacenters,” in Proceedings of the 2nd ACM

Symposium on Cloud Computing, ser. SOCC ’11. New York,NY, USA: ACM, 2011, pp. 19:1–19:14.

[15] S. Kraft, G. Casale, D. Krishnamurthy, D. Greer, and P. Kil-patrick, “Io performance prediction in consolidated virtualizedenvironments,” SIGSOFT Softw. Eng. Notes, vol. 36, no. 5, pp.295–306, Sep. 2011.

[16] M. Abd-El-Malek, W. V. Courtright, II, C. Cranor, G. R.Ganger, J. Hendricks, A. J. Klosterman, M. Mesnier,M. Prasad, B. Salmon, R. R. Sambasivan, S. Sinnamohideen,J. D. Strunk, E. Thereska, M. Wachs, and J. J. Wylie, “Ursaminor: versatile cluster-based storage,” in Proceedings of the

4th conference on USENIX Conference on File and Storage

Technologies - Volume 4, ser. FAST’05. Berkeley, CA, USA:USENIX Association, 2005, pp. 5–5.

[17] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati,A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall,and W. Vogels, “Dynamo: amazon’s highly available key-valuestore,” SIGOPS Oper. Syst. Rev., vol. 41, no. 6, pp. 205–220,Oct. 2007.

[18] A. Lakshman and P. Malik, “Cassandra: a decentralized struc-tured storage system,” SIGOPS Oper. Syst. Rev., vol. 44, no. 2,pp. 35–40, Apr. 2010.

[19] T. Schutt, F. Schintke, and A. Reinefeld, “Scalaris: reliabletransactional p2p key/value store,” in Proceedings of the 7th

ACM SIGPLAN workshop on ERLANG, ser. ERLANG ’08,2008, pp. 41–48.

8