Fiber Cabling Trends in Data Centersgrouper.ieee.org/groups/802/3/100GNGOPTX/public/... · Fiber...

1

Fiber Cabling Trends in Data Centers

Paul Kolesar, CommScopeIEEE 802.3

Next Generation Optics Study GroupSeptember, 2011

2

Objective & Methodology

• Examine fiber cabling channel length distributions over time to identify trends that will allow informed determination of reach targets

• Extract length distributions from CommScope sales data base of factory-terminated fiber cabling assemblies used in building data centers

Disclosure: Similar material was contributed toEthernet Alliance in May 2011, andINCITS T11.2 in August 2011

3

Outline

• Architecture and cabling terminology• Fiber type trends• Generating channel length distributions

– Cords– Permanent Links– Channels

• Reach coverage• Channel length trends• Future Work• Summary

4

Data Center Architecture Terminology

source: IEEE 802.3 contribution barbieri_01_0107.pdf

5

Cabling TerminologyCords – used to administer connectivity from:1. equipment (transceivers) to patch panel, called equipment cords2. patch panel to patch panel, called patch cords

Links – permanent cabling between two patch panels,built in Data Centers primarily using 12-fiber cable subunits

Channels – complete end-to-end connectivity between equipment consisting of concatenations of cords and permanent link(s)

perm. linkequip. cord

Single-link channels

xcvr xcvr

equip. cord

Double-link channels

xcvr xcvr

equip. cord equip. cordpatch cordperm. link perm. link

6

Fiber Media Mix 2008 - 2010

• Laser-optimized 50 µm dominated (~85%)

• Single-mode variedbetween 10 – 15%

• 62.5 µm faded to insignificance

Fiber Mix by Total 12-fiber Subunits

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2008 2009 2010

62.5umSM50um

7

single cord length distribution

0%

5%

10%

15%

20%

25%

30%

35%

40%

0-56-10

11-15

16-20

21-25

26-30

31-35

36-40

41-45

46-50

51-55

56-60

61-65

66-70

71-75

76-80

81-85

86-90

length (ft)

Prob

abili

ty

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Pro

babi

lity

mean = 12.6 ftstd.dev. = 12.2 ft

Truncatedabove 70 ftto remove outliers.Eliminatesupper 1.4%.

Cord Distributions

Convolved twice with

itself

Convolved once withitself

For 1-link channels(containing 2 cords)

For 2-link channels(containing 3 cords)

Length distribution of 2 concatenated cords

0%

5%

10%

15%

20%

25%

30%

35%

40%

1-10

11-20

21-30

31-40

41-50

51-60

61-70

71-80

81-90

91-10

010

1-110

Length (ft)

Prob

abab

ility

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%Cu

mul

ativ

e P

roba

bilit

y



0%

5%

10%

15%

20%

25%

30%

35%

40%

1-15

16-30

31-45

46-60

61-75

76-90

91-10

510

6-120

121-1

3513

6-150

Length (ft)

Pro

baba

bilit

y

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Prob

abili

ty


8

Permanent Link Distributions 2008 - 2010

2008 - 2010 MPO permanent link length distributions(equates to distance between patch panels)

0.0%

2.5%

5.0%

7.5%

10.0%

12.5%

15.0%

17.5%

20.0%

22.5%

25.0%

0-30 31-60 61-90 91-120 121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

> 390ft

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

200820092010200820092010

overall mean = 134 ftoverall std.dev. = 120 ftmean variation = +/- 8 ftstd.dev. variation = +/- 4 ft

Trend?

9

Permanent Link Concatenation

Convolved once with

itself.

Yields 2-linkdistribution.

Concatenation ratio = 1

2010 MPO trunk length distribution

0%

5%

10%

15%

20%

25%

0-30 31-60

61-90

91-120

121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

>390ft

Length (ft)

Prob

abili

ty

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum

ulat

ive

Pro

babi

lity

2010 permanent link (w/o cords) length distribution for two concatenated links (constituent of 100% concatenated channel mix)

0%

2%

4%

6%

8%

10%

12%

14%

16%

0-30 31-60

61-90

91-120

121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

391-420

421-450

451-480

481-510

511-540

541-570

571-600

601-630

631-660

>660ft

Length (ft)

Prob

abili

ty

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Prob

abili

ty

10

Channel Development

=

1-link and 2-cord convolution (channel concatenation ratio = 0.0)

*

1 link 2 cords Single-link channels

xcvr xcvr2010 MPO trunk length distribution

0%

5%

10%

15%

20%

25%

0-30 31-60

61-90

91-120

121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

>390ft

Length (ft)

Prob

abili

ty

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum

ulat

ive

Pro

babi

lity

2010 channel (w cords) length distribution for one link (0% concatenated channels)

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

0-30 31-60

61-90

91-120

121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

391-420

421-450

451-480

481-510

511-540

541-570

571-600

601-630

631-660

>660ft

Length bins (ft)

Prob

abili

ty

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum

ulat

ive

Prob

abili

ty


0%

5%

10%

15%

20%

25%

30%

35%

40%

1-10

11-20

21-30

31-40

41-50

51-60

61-70

71-80

81-90

91-10

010

1-110

Length (ft)

Pro

baba

bilit

y

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Pro

babi

lity


* =

2-link and 3-cord convolution (channel concatenation ratio = 1.0)2 links 3 cords Double-link channels

xcvr xcvr2010 permanent link (w/o cords) length distribution

for two concatenated links (constituent of 100% concatenated channel mix)

0%

2%

4%

6%

8%

10%

12%

14%

16%

0-30 31-60

61-90

91-120

121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

391-420

421-450

451-480

481-510

511-540

541-570

571-600

601-630

631-660

>660ft

Length (ft)

Prob

abili

ty

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Prob

abili

ty

2010 channel (w cords) length distribution for two concatenated links (100% concatenated channels)

0%

2%

4%

6%

8%

10%

12%

14%

16%

0-30

31-60

61-90

91-12

012

1-150

151-1

8018

1-210

211-2

4024

1-270

271-3

0030

1-330

331-3

6036

1-390

391-4

2042

1-450

451-4

8048

1-510

511-5

4054

1-570

571-6

0060

1-630

631-6

6066

1-690

691-7

2072

1-750

751-7

8078

1-810

811-8

4084

1-870

871-9

00>90

0 ft

Length bins (ft)

Pro

babi

lity

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Pro

babi

lity


0%

5%

10%

15%

20%

25%

30%

35%

40%

1-15

16-30

31-45

46-60

61-75

76-90

91-10

510

6-120

121-1

3513

6-150

Length (ft)

Prob

abab

ility

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Prob

abili

ty


11

Channel Mixture Example

=+⅔ ⅓

2:1 channel mixture (channel concatenation ratio = 0.33)

Single-link channels

2:1 composite mix of channels

Double-link channels2010 channel (w cords) length distribution for one link (0% concatenated channels)

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

0-30 31-60

61-90

91-120

121-150

151-180

181-210

211-240

241-270

271-300

301-330

331-360

361-390

391-420

421-450

451-480

481-510

511-540

541-570

571-600

601-630

631-660

>660ft

Length bins (ft)

Prob

abili

ty

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum

ulat

ive

Prob

abili

ty

2010 channel (w cords) length distribution for two concatenated links (100% concatenated channels)

0%

2%

4%

6%

8%

10%

12%

14%

16%

0-30

31-60

61-90

91-12

012

1-150

151-1

8018

1-210

211-2

4024

1-270

271-3

0030

1-330

331-3

6036

1-390

391-4

2042

1-450

451-4

8048

1-510

511-5

4054

1-570

571-6

0060

1-630

631-6

6066

1-690

691-7

2072

1-750

751-7

8078

1-810

811-8

4084

1-870

871-9

00>90

0 ft

Length bins (ft)

Pro

babi

lity

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Pro

babi

lity

2010 channel (w cords) length distribution for 50% concatenated links (33% concatenated channels)

0%

2%

4%

6%

8%

10%

12%

14%

16%

0-30

31-60

61-90

91-12

012

1-150

151-1

8018

1-210

211-2

4024

1-270

271-3

0030

1-330

331-3

6036

1-390

391-4

2042

1-450

451-4

8048

1-510

511-5

4054

1-570

571-6

0060

1-630

631-6

6066

1-690

691-7

2072

1-750

751-7

8078

1-810

811-8

4084

1-870

871-9

00>90

0 ft

Length (ft)

Pro

babl

ity

0.0%

12.5%

25.0%

37.5%

50.0%

62.5%

75.0%

87.5%

100.0%

Cum

ulat

ive

Pro

babi

lity

12

Channel Length Distributions

Coverage topology sensitivity = (single-link coverage - double-link coverage) / single-link coverage

2010 channel length distributions by topology mix

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

30 60 90 120

150

180

210

240

270 300

330

360

390

420

450 480

510

540

570

600

630 660 690

720

750

780

810

840

870

900

> 900

length (ft)

single-link channels (concat ratio = 0)

2:1 mixed-link channels (concat ratiio = 0.33)

double-link channels (concat ratio = 1.0)

50 m

100 m

150 m

200 m

extreme very high high moderate low

>50% 25-50% 12-25% 6-12% <6%

Coveragetopologysensitivity:

Flatman_01_0108 channels

14

Reach Coverage of Channels

Important region. Recall reading recent report that showsthis represents aggregate mix. Also this mix of 2005 CommScope data best aligned with Flatman survey of 2007.

Reach coverage of channels (w cords) built in 2010 as a function of topology mix

0

25

50

75

100

125

150

175

200

225

250

275

300

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Concatenated Channel Ratio

PMD

Rea

ch (m

)

mean 1 sigma 2 sigma 3 sigma

all single-linktopology

all two-linktopology

~63%

~99%

~96%

~94%

~87%

~86%~65%

~98%

~64%

~84%

~94%

~98%

~92% @ 150 m

~79% @ 100 m

~45% @ 50 m ~13% @ 50 m

~97% @ 200 m

15

Comparison of Channel Lengths 2005ª vs 2010

Comparison of channels built in 2005 and 2010 as a function of topology mix

0

25

50

75

100

125

150

175

200

225

250

275

300

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0


PMD

Rea

ch (m

)

'10 mean '10 1-sigma '10 2-sigma '10 3-sigma'05 mean '05 1-sigma '05 2-sigma '05 3-sigma

up 13m

up 20m

up 27m

up 34m



a. sources: IEEE 802.3 contributions Kolesar_1_0906.pdf and kolesar_01_0308.pdf

16

Possible Explanation for Length Increase

• Server form migration trend: pedestal ► rack ►blade

– Server density increase drives localization of access switch

– Single server rack can consume switch

• Access switch placement migration: centralized ► end-of-row ► top-of-rack

– Shortening copper access channels lengthens fiber aggregation channels

Core

Servers

Access

Distribution

Core

top-of-rack

Server

Access

Distribution

centralized,end-of-row

migration

Server

Server

fiber

rack,blade

pedestal

copper bladerackpedestal

Sources: IDC, Sep. 2010IEEE 802.3 CFI_01_1110.pdfflatman_01_0311.pdf

17

Future Work

• Estimate solution costs by examiningtechnologies vs. reach coverage

– MM and SM optics– Retiming– Equalization– Encoding– Dispersion reduction

• Calculate optimal combinations that give lowest overall data center cost

– Calculator for PMDs prepared for use – Input reaches and relative costs for PMD solution set members

• (e.g. -CR4, -SR4, -LR4)– Outputs relative cost for each topology mix on slide 12

Great topics for this study group

18

Summary

• Fiber types have evolved– Laser-optimized 50 µm now dominates– SM minor share varies, 62.5 µm faded away

• Channel lengths have increased– Up an average of 13 m at 0.33 concatenation ratio– Follows access switch localization to server

• Reach coverage became more topology dependent– Advisable to strive for sufficient reach

to achieve moderate to low sensitivity– Examine technology trade-offs

to optimize overall cost of DC solutions

Fiber Mix by Total 12-fiber Subunits

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2008 2009 2010

62.5umSM50um

Comparison of channels built in 2005 and 2010 as a function of topology mix

0

25

50

75

100

125

150

175

200

225

250

275

300

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0


PMD

Rea

ch (m

)

'10 mean '10 1-sigma '10 2-sigma '10 3-sigma'05 mean '05 1-sigma '05 2-sigma '05 3-sigma

up 13m

up 20m

up 27m

up 34m



2010 channel length distributions by topology mix

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

30 60 90 120

150

180

210

240

270

300

330

360

390

420

450

480

510

540

570

600

630

660

690

720

750

780

810

840

870

900

> 900

length (ft)

single-link channels (concat ratio = 0)

2:1 mixed-link channels (concat ratiio = 0.33)

double-link channels (concat ratio = 1.0)

50 m

100 m

150 m

200 m

extreme very high high moderate low

>50% 25-50% 12-25% 6-12% <6%

Coveragetopologysensitivity:

19

Thank You

Fiber Cabling Trends in Data Centersgrouper.ieee.org/groups/802/3/100GNGOPTX/public/... · Fiber...

Documents

Transcript of Fiber Cabling Trends in Data Centersgrouper.ieee.org/groups/802/3/100GNGOPTX/public/... · Fiber...