Distributed Algorithms – 2g1513 Lecture 10 – by Ali Ghodsi Fault-Tolerance in Asynchronous Networks.
Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi,...
-
Upload
odalys-nazworth -
Category
Documents
-
view
216 -
download
2
Transcript of Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi,...
![Page 1: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/1.jpg)
Aggressive Cloning of Jobs for Effective Straggler Mitigation
Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica
![Page 2: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/2.jpg)
Small jobs increasingly important
• Most jobs are small– 82% of jobs contain less than 10 tasks (Facebook’s
Hadoop cluster)
• Most small jobs are interactive and latency-constrained– Data analyst testing query on small sample
• Small jobs particularly sensitive to stragglers
![Page 3: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/3.jpg)
Straggler Mitigation
• Blacklisting: – Clusters periodically diagnose and eliminate
machines with faulty hardware
• Speculation: Non-deterministic stragglers– Complete systemic modeling is intrinsically complex
[e.g., Dean’12 at Google]– LATE [OSDI’08], Mantri [OSDI’10]…
![Page 4: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/4.jpg)
Despite the mitigation techniques…
LATE: The slowest task runs 8 times slower than the median task
Mantri: The slowest task runs 6 times slower than the median task
• (…but they work well for large jobs)
![Page 5: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/5.jpg)
State-of-the-Art Straggler Mitigation
Speculative Execution:(in LATE, Mantri, MapReduce)
1. Wait: observe relative progress rates of tasks
2. Speculate: launch copies of tasks that are predicted to be stragglers
![Page 6: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/6.jpg)
Why doesn’t this work for small jobs?
1. Consist of just a few tasks– Statistically hard to predict stragglers– Need to wait longer to accurately predict stragglers
2. Run all their tasks simultaneously– Waiting can constitute considerable fraction of a
small job’s duration
Wait & Speculate is ill-suited to address stragglers in small jobs
![Page 7: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/7.jpg)
Cloning Jobs
• Proactively launch clones of a job, just as they are submitted
• Pick the result from the earliest clone• Probabilistically mitigates stragglers
• Eschews waiting, speculation, causal analysis…
Is this really feasible??
![Page 8: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/8.jpg)
Low Cluster Utilization
• Clusters have median utilization of under 20%– Provisioned for (short burst of) peak utilization
• Cluster energy-efficiency proposals– Not adopted in today’s clusters! – Peak utilization decides half the energy bill– Hardware and software reliability issues…
![Page 9: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/9.jpg)
Power-law exponent = 1.9
Tragedy of commons?
Power-law: 90% of jobs use
6% of resources FB, Bing, Yahoo!
Can clone small jobs with few extra resources
• If every job utilizes the “lowly utilized” cluster…– Instability and negative performance effects
![Page 10: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/10.jpg)
Job
Strawman
Earliest
Easy to implement Directly extends to any framework
M1M2
M2
R1
R1M1
![Page 11: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/11.jpg)
Number of map clones
• Contention for input data by map task clones
• Storage crunch Cannot increase replication
>> 3 clones
![Page 12: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/12.jpg)
Task-level Cloning
Job
Earliest
M1M1
M2
R1
R1M2
Earliest Earliest
![Page 13: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/13.jpg)
≤3 clones sufficesStrawman Task-level Cloning
![Page 14: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/14.jpg)
Dolly: Cloning Jobs
• Task-level cloning of jobs• Works within a budget– Cap on the extra cluster resources for cloning
![Page 15: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/15.jpg)
Evaluation
• Workload derived from Facebook traces– FB: 3500 node Hadoop cluster, 375K jobs, 1 month
• Trace-driven simulator
• Baselines: LATE and Mantri, + blacklisting• Cloning budget of 5%
![Page 16: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/16.jpg)
Baseline: LATESmall jobs benefit significantly!
Average completion time improves by 44%
![Page 17: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/17.jpg)
Baseline: MantriSmall jobs benefit significantly!
Average completion time improves by 42%
![Page 18: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/18.jpg)
Intermediate Data Contention
• We would like every reduce clone to get its own copy of intermediate data (map output)
• Not replicated, to avoid overheads
• What if a map clone straggles?
![Page 19: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/19.jpg)
Intermediate Data Contention
M1
M1
M2
M2
R1
R1
M1
M2
Wait for exclusive copy or contend for the available copy?
![Page 20: Aggressive Cloning of Jobs for Effective Straggler Mitigation Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, Ion Stoica.](https://reader036.fdocuments.in/reader036/viewer/2022062511/551b0eff550346cf5a8b4f3f/html5/thumbnails/20.jpg)
Conclusion
• Stragglers in small jobs are not well-handled by traditional mitigation strategies– Guessing task to speculate very hard, waiting wastes
significant computation time
• Dolly: Proactive Cloning of jobs– Power-law Small cloning budget (5%) suffices– Jobs improve by at least 42% w.r.t. state-of-the-art
straggler mitigation strategies
• Low utilization + Power-law + Cloning?