1

High PerformanceSwitching and RoutingTelecom Center Workshop: Sept 4, 1997.

Maintaining Packet Order in Two-Stage Switches

Isaac Keslassy, Nick McKeownStanford University

2

Two Successive Scaling Problems

OQ routers: + work-conserving- memory bandwidth =

(N+1)RR

R

RR

IQ routers: + memory bandwidth = 2R- arbitration complexity

Bipartite Matching

R R

3

Today: 64 ports at 10Gbps, 64-byte cells.

• Arbitration Time = = 51.2ns

• Request/Grant Communication BW = 17.5Gbps

10Gbps 64bytes

IQ Arbitration Complexity

Two main alternatives for scaling:1. Increase cell size (Kar et al., 2000)2. Eliminate arbitration (Chang et al., 2001)

Scaling to 160Gbps:• Arbitration Time = 3.2ns• Request/Grant Communication BW = 280Gbps

4

Desirable Characteristics for Router Architecture

Ideal: OQ• 100% throughput• Minimum delay• Maintains packet order

Necessary: able to regularly connect any input to any output

What if the world was perfect? Assume Bernoulli iid uniform arrival traffic...

5

Cyclic Shift?

1

N

1

N

Uniform Bernoulli iid traffic: 100% throughput

6

Cyclic Shift?

1

N

1

N

b(t) (t)q(t)

Problem: real traffic is non-uniform

01

1)()(

1lim

e

N

T

tttb

TT

Long-term service opportunities exceed arrivals:

1

)]([ eN

tbE

7

Two-Stage Switch

External Outputs

Internal Inputs

1

N

ExternalInputs

Load-balancing cyclic shift

Switching cyclic shift

1

N

1

N

11

2

2

100% throughput for broad range of traffic types (C.S. Chang et al., 2001)

8

Two-Stage SwitchExternal Outputs

Internal Inputs

1

N

ExternalInputs

1

N

1

N

1(t) 2(t)

b(t)

q(t)

a(t)

• Traffic rate: ΛeN

taEtEtatEtbE1

)]([)]([)]()([)]([ 11 ππ

)()()( 1 tattb π• First cyclic shift:

011

1)()(

1lim 2

e

Ne

N

T

tttb

TT

• Long-term service opportunities exceed arrivals:

• (2 = 1 possible)

9

• Eliminates arbitration

• 100% throughput

• Conventional router packaging

Two-Stage Switch Characteristics

10

1

2

3

Phase 2

Phase 1

Using a Single Stage Twice

Lookup

Buffer

Lookup

Buffer

Lookup

Buffer

Linecards

1

N

1

N

1

N

11

Racks of linecards

Optical links

Optical SwitchFabric

Two-Stage Switch Characteristics

12

Problem: unbounded mis-sequencing

External Outputs

Internal Inputs

1

N

ExternalInputs

Cyclic Shift Cyclic Shift

1

N

1

N

11

2

2

Two-Stage Switch Characteristics

13

Full Frames First (FFF):Intuitive Idea

External Outputs

Internal Inputs

1

N

ExternalInputs

Cyclic Shift Cyclic Shift

1

N

1

N

11

2

2

Idea: 1. Spread cells evenly across all linecards

t1 2

14

Full Frames First (FFF):Intuitive Idea

Idea: 1. Spread cells evenly across all linecards2. Read them in order

External Outputs

Internal Inputs

1

N

ExternalInputs

Cyclic Shift Cyclic Shift

1

N

1

N

11

2

3

2

15

Full Frames First (FFF):Intuitive Idea

Idea: 1. Spread cells evenly across all linecards2. Read them in order

External Outputs

Internal Inputs

1

N

ExternalInputs

Cyclic Shift Cyclic Shift

1

N

1

N

1

2

2

3

1

16

First Problem

Problem: if two packets don’t arrive consecutively, there may be a hole in the reading sequence

External Outputs

Internal Inputs

1

N

ExternalInputs

Cyclic Shift Cyclic Shift

1

N

1

N

11

2

2

3

t1 2 3

17

Coordination Buffer

Solution: collect cells from a flow in a coordination buffer, and load-balance them among linecards

1

N

Cyclic Shift Cyclic Shift

1

N

1

N

12

3

2

Flow Load Balancing

Coordination Buffer (VOQ)

1

t1 2 3

18

Second Problem

Problem: No access to cell 2 because of head-of-line blocking

1

N

Cyclic Shift Cyclic Shift

1

N

1

N

12

2

Flow Load Balancing

Coordination Buffer (VOQ)

1

3

a

b

t1 32

ta b

Input 1:Input 2:

19

Expanding VOQ Structure

Solution: expand VOQ structure by distinguishing among switch inputs

2

1

3

a

b

20

FFF: Guarantees

Theorem 2: for any arrival process, Davg (FFF) Davg (OQ) + (4N2 - 2)

Theorem 1: for any arrival process for which OQ has 100% throughput, so does FFF

Theorem 3: FFF maintains packet order

21

Two-Stage Switch in Optics

R/N

Passive mesh

1

23

1

23

R/NR/N

Passive mesh

1

23

1

23

2R/N

123

123

Cyclic Shift Cyclic Shift

R R

22

Summary

• FFF: practical algorithm that solves mis-sequencing

• Same throughput as OQ, and average delay within a bound

• New approach to optical switching