Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture P. J. García 1,...

Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture

P. J. García1, J. Flich2, J. Duato2, I. Johnson3, F. J. Quiles1, F. Naven3

2Technical University of Valencia

Valencia, Spain

3Xyratex

Havant, UK

1University of Castilla-La Mancha

Albacete, Spain

HiPEAC 2005 17 November - 18 November Barcelona, Spain

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

2

Outline

• Introduction

• Congestion trees and HOL blocking

• HOL blocking elimination techniques

• Traditional view of congestion trees

• Different dynamics of congestion trees

• RECN improvements

• Performance evaluation

• Conclusions

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

3

Introduction

High-speed interconnection networks:• Myrinet, Infiniband, Quadrics, Advanced Switching…• Main features: High bandwidth, Low latencies• Additional features: Lossless networks, Flexible topology• Cost and power consumption considerations

recommend working close to the saturation point

Network performance may be affected by congestion

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

4

Contention:• Several packets request the same output port• One makes progress, the others wait

Congestion:• Persistent contention • It is quickly propagated by flow control (lossless nets),

forming congestion trees• Network performance degrades dramatically!!!

Congestion trees and HOL blocking

Head of line (HOL) blocking:• When the first packet in a queue is blocked, any other

packet in the same queue is also blocked, even if it will request available resources

WHY?

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

5


Networkcontention

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

6


Persistentnetworkcontention

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

7



Flow control

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

8



Congestionpropagates

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

9


Congestiontree root

Congestiontree leaf

Congestiontree leaf

Congestiontree branch

Congestiontree branch

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

10


Congestion trees introduce HOL blocking, and this may degrade network performance dramatically

33%

33%

HOL 33%

33%100%

33%

33%

33%

100%

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

11

HOL blocking elimination/reduction techniques

• DAMQs and Virtual Channels • Different buffers for different flows

• VOQ (Virtual Output Queues)• VOQ at switch level: A separate queue at every input port for every

output port• VOQ at network level: A separate queue at every input port for every

destination

• Credit Flow Controlled ATM• Handles congestion at network outputs only• A separate queue at every output port for every destination

In general, these techniques try to separate different flows of packets in order to avoid HOL blocking:

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

12

RECN: Regional Explicit Congestion Notification

• RECN is a new efficient and scalable congestion management technique

• Basic ideas:• The real problem is not the congestion, but its negative effects

(HOL blocking)• By eliminating HOL blocking, congestion becomes harmless• Non-congested flows do not introduce significant HOL blocking

• HOL blocking elimination: • Packets belonging to congested flows are stored in specific Set

Aside Queues (SAQs)• Packets belonging to non-congested flows are stored in a

“common” queue

• Implementation requirements:• Deterministic source routing• A reduced number of SAQs per port, controlled by a CAM

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

14

A congestion point forms

How RECN Works

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

15

How RECN Works

Cold queue fills over a threshold

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

17

How RECN Works

Internal notification to each input port

sending packets to the output port

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

19

How RECN Works

Input ports allocate a new SAQ for

packets addressed tothe congested output port

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

21

How RECN Works

Notification sent whenthe SAQ fills

over a threshold

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

23

How RECN Works

A new SAQ allocatedfor the congested port

at each output port

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

24

How RECN Works

Internal notification when the SAQ fills over

A threshold

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

25

How RECN Works

The input port allocatesA new SAQ

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

26

How RECN Works

At the end, the congestion tree builds and is mapped

entirely onto SAQs

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

27

Traditional view of congestion trees

Traditional ideas about congestion trees growth:• Congestion propagates from the root to the leaves• Congestion first appears at egress sides

This is not always true: Congestion trees may evolve in several ways

The effectiveness of HOL blocking elimination techniques may drop if they do not consider

congestion tree dynamics

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

28

Different dynamics of congestion trees

Effect of switch architecture (I):• Switch speedup may vary for different technologies• Depending on switch speedup, congestion may

appear at ingress or egress sides

No speedup switch

Full rateinjection Congestion

Switch speedup: 2

Congestion

Full rateinjection Congestion

Switch speedup: 2

Full rateinjection

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

29

• Switch speedup: 2• All the sources start

injection simultaneously


Effect of switch architecture (II):• Several congested points may appear both at

ingress or egress sides along the branches of a congestion tree

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

30


Impact of traffic patterns (I):• Depending on traffic patterns, the congestion tree

root may “move” downstream

• Switch speedup: 2• Solid flows appear first, dashed

ones later

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

31


Impact of traffic patterns (II):• Different congestion trees may merge, even when

the involved packets have different destinations


ones later

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

32


Impact of traffic patterns (III):• Different congestion trees may overlap without

merging


ones later

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

33


Impact of traffic patterns (IV):• A congestion tree root may also move upstream

• Switch speedup: 2• dashed flow disappears first

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

36

RECN improvements

Modified (“Enhanced”) RECN:• Congestion is detected at ingress or egress ports

– Ingress cold queues are replaced by small “detection queues”, one per output port

– If a detection queue fills over a threshold, congestion is detected for the corresponding output port

• It is allowed the allocation of more-specific SAQs– In order to keep in-order delivery of packets, a new

allocated and more-specific SAQ is blocked until all the packets on the less-specific SAQ are forwarded

– A pointer to the new SAQ is placed on the less-specific SAQ in order to control the blocking

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

38

Performance Evaluation

•Objective: Evaluation of RECN improvements

•Comparative evaluation based on simulation results

•Evaluation metric:• Network throughput when using:

– Basic RECN– Enhanced RECN– VOQ at switch level (VOQsw)

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

39

Simulation Model

• Network configurations evaluated:• 64 hosts connected by a 64x64 BMIN• 512 hosts connected by a 512x512 BMIN• 2048 hosts connected by a 2048x2048 BMIN

• Simulation assumptions:• BMINs based on perfect shuffle scheme• Deterministic routing• 32 KB memories at ingress/egress ports• Multiplexed crossbar (BW=8 or12 Gbps)• Serial full-duplex pipelined links (BW=8 Gbps)• 64-byte packets• Credit-based and Xon-Xoff (for SAQs) flow control• Maximum of 8 SAQs at ingress/egress ports (RECN)

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

40

Traffic Load

• Six different synthetic traffic patterns:

•Traces:• From I/O activity at cello system disk interface• A compression factor applied

Normal traffic Congestion tree

Traffic case

Endnodes #Sources Dest.Injection

rate#Sources Dest.

Injection rate

Congestion type

#1 64x64 75% Rand. 50% 25%Single

hot-spot100% Incremental

#2 64x64 75% Rand. 100% 25%Single

hot-spot100% Incremental

#3 64x64 75% Rand. 50% 25%Single

hot-spot100% sudden

#4 64x64 75% Rand. 100% 25%Single

hot-spot100% sudden

#5 512x512 75% Rand. 100% 25%Four

hot-spot100% sudden

#6 2048x2048 75% Rand. 100% 25%Four

hot-spots100% sudden

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

41

Simulation Results

•Network throughput:• Traffic cases 1 and 2 (single hot-spot incremental traffic) • 64-endnodes networks• Speedup: 1.5

Traffic case 1

(Uniform traffic injection rate 50%)

Traffic case 2


Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

42

Simulation Results

•Network throughput:• SAN traffic (traces) • 64-endnodes networks• Traces compression factor: 40

Speedup 1.5No Speedup

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

43

Simulation Results

•Network throughput:• Traffic cases 3 and 4 (single hot-spot sudden traffic) • 64-endnodes networks• No Speedup

Traffic case 3


Traffic case 4


Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

44

Simulation Results

•Network throughput:• Traffic cases 3 and 4 (single hot-spot sudden traffic) • 64-endnodes networks• Speedup: 1.5

Traffic case 3


Traffic case 4


Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

45

Simulation Results

•Network throughput:• Traffic cases 5 and 6 (four hot-spots sudden traffic) • Uniform traffic injection rate 100%• Speedup: 1.5

Traffic case 5

(512-endnodes network)

Traffic case 6

(2048-endnodes network)

Tit

le:

Dynam

ic E

volu

tion o

f C

on

gest

ion T

rees:

Analy

sis

and Im

pact

on S

wit

ch A

rchit

ect

ure

Confe

rence

: H

iPEA

C 2

00

51

7-1

8 N

ovem

ber

Barc

elo

na, S

pain

46

Conclusions

• Congestion trees producing HOL blocking may affect network performance

• We have shown that congestion trees may form and evolve in different ways

• We have analyzed the importance of considering congestion trees dynamics on the design of HOL blocking elimination techniques

• We have proposed some improvements for RECN, in order to manage HOL blocking independently of the way congestion trees form

• From the results of our experiments, these improvements were necessary

Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture

P. J. García1, J. Flich2, J. Duato2, I. Johnson3, F. J. Quiles1, F. Naven3

2Technical University of Valencia

Valencia, Spain

3Xyratex

Havant, UK

1University of Castilla-La Mancha

Albacete, Spain

HiPEAC 2005 17 November - 18 November Barcelona, Spain

Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture P. J. García 1,...

Documents

Transcript of Dynamic Evolution of Congestion Trees: Analysis and Impact on Switch Architecture P. J. García 1,...