Towards Optimizing the Network for Distributed Machine Learning · 2019-12-18 · Network Core...

Optimizing Network Performance in Distributed Machine Learning

Luo Mai Chuntao Hong Paolo Costa

Machine Learning

• Successful in many fields• Online advertisement

• Spam filtering

• Fraud detection

• Image recognition

• …

• One of the most important workloads in data centers

2

Industry Scale Machine Learning

• More data, higher accuracy

• Scales of industry problems• 100 Billions samples, 1TBs – 1PBs data

• 10 Billions parameters, 1GBs – 1TBs data

• Distributed execution • 100s – 1000s machines

3

Distributed Machine Learning

Data partitions

Model replicas

Workers

W1 W2 W3 W4 Data partitions


Data partitions

Model replicas

Workers5

W1 + 0.1 W2 + 0.2 W3 – 0.3 W4 +1.2 W1– 0.9 W2 + 0.5 W3 – 0.1 W4 – 0.5

gradient


Data partitions

Model replicas

Workers

1. Push gradients

Parameter server

6

2. Aggregate gradient for each parameter


Data partitions

Model replicas

Workers

Parameter server

3. Add gradients to parameters

4. Pull new parameters

7

W1 + g1 W2 + g2 W3 + g3 W4 + g4


Data partitions

Model replicas

Workers

Parameter servers

8

W3 W4W1 W2

Use multiple PS to avoid

bottleneck


Data partitions

Model replicas

Workers

Parameter servers

Bottleneck

9

Inbound Congestion

10

Network Core

Inbound congestion

Outbound Congestion

11

Outbound congestion

Network Core

Network Core Congestion

12

Over-subscribed Network Core

Congestion in the core in case of over-subscribed

networks

Existing Approaches

13

• Over-provisioning networkExpensive

Limited deployment scale

Not available in public cloudsTraining algorithm

Fast network H/We.g., Infiniband and

RoCE

Existing Approaches

14

• Over-provisioning networkExpensive

Limited deployment scale

Not available in public Clouds

• Asynchronous training algorithmTraining efficiency

Might not converge

Asynchronoustraining algorithm

Network H/W

Rethinking the Network Design

Training algorithm

Network H/W

MLNet

15

MLNet is a communication layerdesigned for distributed machinelearning systems

Improves communication efficiency

Orthogonal to existing approaches

Rethinking the Network Design

Training algorithm

Network H/W

MLNet

16

MLNet is a communication layerdesigned for distributed machinelearning systemsImproves communication efficiency

Orthogonal to existing approaches

Optimizations:Traffic reduction

Flow prioritization

Traffic Reduction

17

18

Aggregate the gradients from 6 workers

𝑔1 = 𝑔11+ 𝑔12+ 𝑔13 + 𝑔14+ 𝑔15+ 𝑔16

Traffic Reduction: Key Insight

Workers

Parameterserver

Aggregation is commutative and associative

19

𝒈𝟏𝟏 + 𝒈𝟏𝟐 +𝒈𝟏𝟑 𝒈𝟏𝟒 + 𝒈𝟏𝟓 +𝒈𝟏𝟔

Aggregate the gradients from 6 workers

Traffic Reduction: Key Insight

Aggregate gradientsincrementally does notchange the final result

Traffic Reduction: Design

20

Intercept the push message from the worker to the PS


21

Redirect the messages to a local worker for partial aggregation


22

Send the partial results to the PS for final aggregation

23

More details on the paper:1. Traffic reduction in pull request2. Asynchronous communication

Traffic Prioritization

24

Traffic Prioritization: Key Insight

25

Job 1 Job 2 Job 3 Job 4

These four TCP flows share a bottleneck link and each of

them gets 25% of its bandwidth


26

0 1 2 3 4

Flow Completion Time (FCT)

Average completion time is 4Model 1 Model 2 Model 3 Model 4

Job 1

Job 2

Job 3

Job 4

All flows are delayed! TCP per-flow fairness is harmful in distributed

machine learning.


27

MLNet prioritizes the competing flows to minimize the average training time

Job 1 Job 2 Job 3 Job 4


28

Job 1

0 1 2 3 4

Job 2

Job 3

Job 4

Flow Completion Time (FCT)

Average completion time is 2Model 1 Model 2 Model 3 Model 4

Shorten average FCT can largely improve average training time

Evaluation

• Simulate common network topology in data centers• Classic 10Gbps 1024-node data center topology [Fat-Tree, SIGCOMM’08]

• Training large scale logistic regression• 65B parameters, 141TB dataset [Parameter Server, OSDI’14]

• 800 workers [Parameter Server, OSDI’14]

• With production trace• Data processing rate: uniform(100, 200) MBps

• Synchronize every 30 seconds

29

30

Traffic Reduction (Non-oversubscribed Net.)

02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)

Number of parameter servers

Rack Baseline

Better

Worse

Cost-effective Expensive

02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)


Rack Baseline

31


Better

Worse


Rack reduces 48% completion time

02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)


Rack Baseline


32

Better

Worse


Deploying more parameter serversresolve edge network bottlenecks

33


02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)


Rack Baseline

Better

Worse


Deploying more parameter servers to reduce training time(1) uses more machines(2) only possible with non-oversubscribed networks

02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)


Rack Baseline

34

Traffic Reduction (1:4 Oversubscribed Net.)

Better

Worse


MLNet reduces congestionin the network core.

Reduces training time by >70%


• 20 jobs running in the same cluster

35

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10 12 14

CD

F

Training time (Hours)

Baseline Prioritization

Everyone finish (almost) at the same time


36

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10 12 14

CD

F


Baseline Prioritization

Improve themedian by 25%

Delay the tail by 2%

Better Worse

Traffic Prioritization + Traffic Reduction

37

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10 12 14

CD

F


Baseline Priori. + Red. Reduction

Improve themedian by 60%

Improve thetail by 54%

Better Worse

38

More details on the paper:1. Binary tree aggregation2. More analysis

Summary

• MLNet can significantly improve the network performance ofdistributed machine learning• Traffic reduction

• Flow prioritization

• Drop-in solution

39

Thanks!

40

Discussion

• Relaxed fault-tolerance?• When worker fails, drop that portion of data

• Adaptive communication• Reduce synchronization when network is busy?

• Hybrid network infrastructure?• Some with 10GE, some with 40GE ROCE, etc.

• Degree of tree?

41


Is the local aggregator a new bottleneck?

42

Example: 15 workers in a rack


Build a balanced aggregation structure such as a binary tree.

43

Example: 15 workers in a rack Binary tree aggregation

44

Traffic Reduction

02468

101214

50 100 200 300 400

Trai

nin

gti

me

(Ho

urs

)


Rack Binary Baseline

Better

Worse


45


02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



Better

Worse


46


02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



Better

Worse


Binary Tree and Rack reduces 78%and 48% completion time


47

02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



Better

Worse


Deploying more parameter serversresolve edge network bottlenecks

48


02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



Better

Worse


Deploying more parameter servers to reduce training time(1) needs more machines(2) only possible with non-oversubscribed networks

49


Better

Worse


02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



50


Better

Worse


MLNet reduces congestionin the network core

02468

101214

50 100 200 300 400

Trai

nin

g ti

me

(Ho

urs

)



51


Better

Worse


Binary is consistently consumingmore bandwidth than Rack

Example: Training a Neural Network

52

Random init weight

Truth: {cat, dog, cat,…}

Calculate error/gradient

W: {w1, w2, w3, w4}

G: {g1, g2, g3, g4}

W’: {w1’, w2’, w3’, w4’}

Update weights

Example: Neural Network

53

Dog : 99%

Cat : 1%

Model

W1

W4

W2

W3

Train Apply

Model Training

54

Model

W1

W4

W2

W3

Refine model

W1

W4

W2

W3

Random Init Final Model

W1

W4

W2

W3

Converge

Towards Optimizing the Network for Distributed Machine Learning · 2019-12-18 · Network Core...

Documents

Transcript of Towards Optimizing the Network for Distributed Machine Learning · 2019-12-18 · Network Core...