Increasing the Performance of

Geo-Distributed Storage Systems

Prof. Dejan Kostić, Network Systems Lab, KTH

WWW.KTH.SE

Geo-Distributed Storage Systems

2

The quality of

replica selection

is key for good

performance

Amazon reports 1% loss in revenue for 100ms latency increase

What is long tail latency and why do we care?

3

99th percentile

Local/Global

Resource Sharing

Queueing

Background Tasks

HW/SW Failures

1% out of millions?

Composite Requests!(bound by slowest sub-request)

What causes it?

Our Approach To Reducing Tail Latencies

4

Measure and analyze

network characteristics

Systematic testing of replica

selection algorithms

Improve latency and load

estimation

Amazon EC2 view from Ireland

5

UDP application level ping. After low pass filter 1 min

Choose

2/3Oregon < Virginia!

Frankfurt

Virginia

Oregon

Why is replica selection hard?

6

Debugging

Testing

Errors(subopt. choices)

Many possible topologies, bandwidth, latency and

loss rates results in a huge testing space

Do not lead to critical failures

Hard to determine the correct behavior

Many potential sources of problems

(sampling, smoothing, selection logic)

GeoPerf [SoCC ‘15]

GeoPerf - a tool for systematic testing of

replica selection algorithms

Uses Symbolic Execution and Lightweight

Modeling/Simulation

Generates a specific set of inputs (latencies)

to expose performance anomalies in a

replica selection algorithm7

7

Symbolic Execution Engine

GeoPerf High-Level

8

Production System

(Cassandra, MongoDB)

Replica Selection

Algorithm

Discrete Event Simulation-A

Algorithm-A

Discrete Event Simulation-B

Algorithm-B

Reference

Replica Selection

Algorithm

Assert

Request Time @A > Request Time @B

Ground Truth

Symbolic Execution Background

9

Declare Symbolic Variable

Evaluate Satisfiability of

Obtained Constraint:

X > 5

Append Constraints:

(X > 5) && (X+Y<20)

(X > 5) && (X+Y<20)&&(X<=10)

Test Case: X = 7

int main(){

int X = symbolic();

int Y = 4;

if (X>5)

if (X+Y<20)

assert(X>10);

return 0;

}

Cassandra bug found by GeoPerf

10

2 close

replicas

1 far

replicawarmed up 20

with samples

if ((first-next)/first>BADNESS_THRESHOLD)

We reported this bug; It has been fixed and

a new test case has been added

Impact of the Bug found (Trace Replay)

● 14 days of latency traces used as an input

● 504 deployments of datacenter triplets

● Cassandra buggy version vs. fixed version

11

Median wasted time for

10% of all deployments

is above 50ms

Latency Estimation Issues

12

Oblivious to the underlying networking architecture(Treat latency as a common stream of samples)

Use a single metric to represent latency(Median, 99th percentile, EWMA etc.)

No insight into the cause of a change(Cannot differentiate between routing and congestion, bad predictors)

Motivation

13

Increasing sensitivity of the median will not solve the problem!

Singapore < VirginiaOscillating behaviorLonger tail

EdgeVar: understand network latency!

The number of network paths between a pair of

geo-distributed datacenters of Amazon EC2 is finite

Packets traveling a previously observed network

path incur the same network delay as previously

observed for this network path

We decompose network latency into routing delay

and residual latency due to competing traffic

We improve the quality of latency estimation, as

well as the stability of network path selection

14

Popular Latency Classes observed

between Ireland and Oregon

15

Each

network path

occurs only

in one class

EdgeVar Latency Decomposition

16

Can easily avoid

congested paths

Use EdgeVar-reported

Baseline latency and

Residual latency to

score network paths

Conclusion

1. GeoPerf: replica selection debugging tool

• Generates latency inputs that can demonstrate

weaknesses in replica selection algorithms

• Found a bug in each of the two popular systems

2. EdgeVar:

• Provides clear understanding of e2e latency :

– level shifts due to routing vs. congestion

17

Thanks: Kirill Bogdanov (KTH), Miguel Peon-Quiros (UCM), Gerald Q.

Maguire Jr. (KTH)

Plan to release source code & latency traces:

https://www.kth.se/blogs/prophet