Sparrow Distributed Low-Latency Scheduling Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion...
43
Sparrow Distributed Low-Latency Scheduling Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion Stoica
-
Upload
kerry-park -
Category
Documents
-
view
219 -
download
1
Transcript of Sparrow Distributed Low-Latency Scheduling Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion...
SparrowDistributed Low-Latency Scheduling
Kay Ousterhout, Patrick Wendell, Matei Zaharia, Ion Stoica
Sparrow schedules tasks in clusters
using a decentralized, randomized approach
Sparrow schedules tasks in clusters
using a decentralized, randomized approach
support constraints and fair sharing
and provides response times
within 12% of ideal
Scheduling Setting
…
…
…
Map Reduce/Spark/
Dryad Job
…
Task
Task
Task
Map Reduce/Spark/
Dryad JobTask
Task
Job Latencies Rapidly Decreasing
10 min.
10 sec.
100 ms
1 ms
2004: MapReducebatch job
2009: Hive
query
2010: Dremel Query
2012: Impala query
2010:In-
memory Spark query
2013:Spark
streaming
Scheduling challenges:
Millisecond Latency
Quality Placement
Fault Tolerant
High Throughput
10 min.
10 sec.
100 ms
1 ms
2004: MapReducebatch job
2009: Hive
query
2010: Dremel Query
2012: Impala query
2010:In-
memory Spark query
2013:Spark
streaming
1000 16-core machines
26decisio
ns/second
Scheduler
throughput
1.6K
decisions/
second
160Kdecisio
ns/second
16M
decisions/
second
Millisecond Latency
Quality Placement
Fault Tolerant
High Throughput
Today: Completely Centralized Less centralization
Sparrow:Completely
Decentralized
✗
✗
✓
✗ ✓
✓
✓
?✓
SparrowDecentralized approach
Existing randomized approachesBatch SamplingLate BindingAnalytical performance evaluation
Handling constraints
Fairness and policy enforcement
Within 12% of ideal on 100 machines
Scheduling with Sparrow
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
Random
Simulated Results
100-task jobs in 10,000-node cluster, exp. task durations
Omniscient: infinitely fast centralized
scheduler
Per-task sampling
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
Power of Two Choices
Per-task sampling
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
Power of Two Choices
Simulated Results
100-task jobs in 10,000-node cluster, exp. task durations
70% cluster load
Response Time Grows with Tasks/Job!
Per-Task Sampling
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
✓
✓
Task 1
Task 2
Per-ta
sk
Per-task Sampling
WorkerWorkerWorkerWorkerWorker
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
Place m tasks on the least loaded of dm slaves
Per-ta
sk✓
✓
4 probes (d =
2)
Batch
Per-task versus Batch Sampling
70% cluster load
Simulated Results
100-task jobs in 10,000-node cluster, exp. task durations
Queue length poor predictor of wait time
Worker
Worker
80 ms155
ms
530 ms
Poor performance on heterogeneous workloads
Late Binding
Worker
Worker
Worker
Worker
Worker
Scheduler
Scheduler
SchedulerSchedulerJob
Worker
Place m tasks on the least loaded of dm slaves
4 probes (d =
2)
Late Binding
Scheduler
Scheduler
SchedulerSchedulerJob
Place m tasks on the least loaded of dm slaves
4 probes (d =
2)
Worker
Worker
Worker
Worker
Worker
Worker
Late Binding
Scheduler
Scheduler
SchedulerSchedulerJob
Place m tasks on the least loaded of dm slaves
Worker
requests
task
Worker
Worker
Worker
Worker
Worker
Worker
Simulated Results
100-task jobs in 10,000-node cluster, exp. task durations
What about constraints?
Job Constraints
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
Worker
Worker
Worker
Worker
Worker
Restrict probed machines to those that satisfy the constraint
Per-Task Constraints
Scheduler
Scheduler
Scheduler
SchedulerJob
Worker
Worker
Worker
Worker
Worker
Worker
Probe separately for each task
Technique Recap
Scheduler
Scheduler
Scheduler
SchedulerBatch
sampling+
Late binding+
Constraint handling
WorkerWorkerWorkerWorkerWorker
Worker
How does Sparrow perform on a real cluster?
Spark on Sparrow
WorkerWorkerWorkerWorkerWorker
Worker
Query: DAG of Stages
Sparrow
Scheduler
Job
Spark on Sparrow
WorkerWorkerWorkerWorkerWorker
Worker
Query: DAG of Stages
Sparrow
Scheduler
Job
Spark on Sparrow
WorkerWorkerWorkerWorkerWorker
Worker
Query: DAG of Stages
Sparrow
Scheduler
Job
How does Sparrow compare to Spark’s native scheduler?
100 16-core EC2 nodes, 10 tasks/job, 10 schedulers, 80% load
TPC-H Queries: Background
TPC-H: Common benchmark for analytics workloads
Sparrow
Spark: Distributed in-memory analytics framework
Shark: SQL execution engine
TPC-H Queries
100 16-core EC2 nodes, 10 schedulers, 80% load
95
75
25
50
Percentiles
5
TPC-H Queries
100 16-core EC2 nodes, 10 schedulers, 80% load
Within 12% of ideal
Median queuing delay of 9ms
Fault Tolerance
Scheduler 1
Scheduler 2
Spark Client 1 ✗Spark
Client 2
Timeout: 100msFailover: 5ms
Re-launch queries: 15ms
When does Sparrow not work as well?
High cluster load
Related Work
Centralized task schedulers: e.g., Quincy
Two level schedulers: e.g., YARN, Mesos
Coarse-grained cluster schedulers: e.g., Omega
Load balancing: single task
Batch sampling
+Late binding
+Constraint handling
www.github.com/radlab/sparrow
Sparrows provides near-ideal job response
times without global visibility
Scheduler
Scheduler
Scheduler
Scheduler
WorkerWorkerWorkerWorkerWorker
Worker
Backup Slides
Can we do better without losing simplicity?
Policy Enforcement
SlaveHigh Priority
Low Priority SlaveUser A (75%)
User B (25%)
Fair SharesServe queues using
weighted fair queuing
PrioritiesServe queues based on strict priorities
