Migration Strategy for 40G/100G Ethernet over40G/100G Ethernet over

Multimode FiberPaul KishBeldenBelden

OutlineOutline• The need for speedp• Evolution of network cabling• 40GBASE SR and 100GBASE SR standard• 40GBASE-SR and 100GBASE-SR standard• Relative economics – 10G vs. 40 G over MMF • Migration from 10G to 40G/100G Ethernet• Some example implementations• Optical loss testing accuracy• Road ahead for optical fiber• Road ahead for optical fiber

THE NEED FOR SPEED40G/100G Ethernet

THE NEED FOR SPEED

Global IP Traffic ForecastGlobal IP Traffic Forecast

Annual global IP traffic Major milestonesAnnual global IP traffic Major milestones

• By 2011 there will be more networked devices than people on earth

• By 2012 Internet video will surpass 50 % of consumer Internet traffic

• By 2015 the annual global IP traffic will reach the zettabyte threshold (966 exabytes)

CAGR 2010 – 2015 = 32%

1 EB = 1 000 000 000 000 000 000

Source: Cisco Visual Networking Index

1 EB = 1,000,000,000,000,000,000

Putting it into PerspectivePutting it into Perspective

byte gigabyte terabyte petabyte exabyte zettabyte8b 000 000 000 000 G 000 000 000

Time to transfer information @ 10 Gbps

1 B = 8b 1,000,000,000 B 1000 GB 1000 TB 1000 PB 1000 EBseconds 1minutes 15

days 10days 10years 30

centuries 300

The amount of information is finite

The scale of the information is staggering

Network Drivers for Higher Speed Ethernet

Multi-core Servers - high-performance compute (HPC) clusters where many servers are networked together to execute large y g gcomputing tasks through parallel processing

Virtualization Trend - consolidating workloads onto fewer, more highly utilized servers using virtualization technologieshighly utilized servers using virtualization technologies

Networked Storage Trend - networked storage shared by multiple servers is replacing the local hard disks in traditional servers

I/O Convergence Trend – servers are beginning to utilize a single converged Ethernet I/O interface to connect to separate storage area networks and local area networks

Data Center Network Aggregation Trend – as the deployment of 10GbE increases, there is a need for higher speed switch uplinks for network aggregation in the data center

* Ethernet Alliance white paper: “Overview of Requirements and Applicationsfor 40 Gigabit and 100 Gigabit Ethernet”

Networking Data RatesNetworking Data Rates40 GbE and 100 GbE Networkingg

EVOLUTION OF NETWORK CABLINGEVOLUTION OF NETWORK CABLING

The Evolution of Multimode Optical Fiber Network Cabling

2005

2010 LO 50 MMF (OM4)

10GBASE SR

40/100GBASE-SR

1995

2000LO 50 MMF (OM3)

50 MMF (OM2)

10GBASE-SR

1000BASE-SX

1990 62.5 MMF (OM1)FDDI

Ethernet Over Multimode Optical Fiber Cabling

SpeedMMF TypeDistance (m)

Name Standard DescriptionDistance (m)

1 GbpsOM1: 275OM2: 550 1000BASE-SX

IEEE 802.3z1998

850 nmVCSEL

2OM3: 8001998 2 MMFs

10 Gbps

OM1: 33OM2: 82

10GBASE-SRIEEE 802.3ae

850 nmVCSEL

10 GbpsOM3: 300OM4: 450

10GBASE SR2002 2 MMFs

40/100 OM3: 100 40GBASE-SR4 IEEE 802.3ba850 nm

40/100 Gbps

OM3: 100OM4: 150

40GBASE SR4100GBASE-SR10

IEEE 802.3ba2010

VCSEL8/20 MMFs

40G and 100G ETHERNET STANDARD40G and 100G ETHERNET STANDARD

40G and 100G Ethernet StandardIEEE Std 802.3ba-2010 standard was approved at the June 2010 IEEE Standards Board meeting. The standard includes:

• Copper40GBASE-CR4: 40 Gb/s over twin-ax copper cabling up to 7 m100GBASE CR10 100 Gb/ i bli 7100GBASE-CR10: 100 Gb/s over twin-ax copper cabling up to 7 m

• Multi mode fiber40GBASE SR4: 40 Gb/s over multi mode fiber up to 150 m40GBASE-SR4: 40 Gb/s over multi mode fiber up to 150 m

100GBASE-SR10: 100 Gb/s over multi mode fiber up to 150 m

• Single mode fiber /40GBASE-LR4: 40 Gb/s over single mode fiber up to 10 km100GBASE-LR4: 100 Gb/s over long single mode fiber up to 10 km100GBASE-ER4: 100 Gb/s over single mode fiber up to 40 km

40GBASE-SR4 Parallel Optics40GBASE SR4 Parallel Optics10GBASE-SR

Tx Rx10 Gb/s

Tx12

1 lane (2 MMF 850 nm)10 Gb/s

Rx

Rx

Tx

40GBASE-SR410 Gb/s

Tx12

Rx

234567R T

10 Gb/s•••

10 Gb/s

789101112

Rx Tx

10 Gb/s•••

13

124 lanes (8 MMF 850 nm)

100GBASE-SR10 Parallel Optics100GBASE SR10 Parallel Optics100GBASE-SR10

T R10 Gb/s

Tx123456

Rx

••

••

10 Gb/s

6789101112

Tx Rx• •

10 Gb/sRx 1

2345

Tx

• ••

10 Gb/s

6789101112

Rx Tx

••

••

14

10 lanes (20 MMF 850 nm)

IEEE 802.3ba Medium Dependent Interface (MDI)40G & 100 G Ethernet Over Multimode Fiber40G & 100 G Ethernet Over Multimode Fiber

Clause 86.10.3.3 Medium Dependent Interface (MDI) requirementsThe MDI adapter shall meet the dimensional specifications of IEC 61754-7p pInterface 7-3, the MPO adapter interface. … The MDI shall optically matewith the plug on the optical fiber cabling.

40GBASE-SR4 & 100GBASE-SR10 Using MPO Interface

40GBASE-SR4 Optical lane assignments

-100GBASE-SR10 Optical lane assignments100GBASE SR10 Optical lane assignments

40GBASE-SR4 Channel Specifications

PMD C i PMDC i

Fiber Optic ChannelMDI MDI

4 lanes

Description Type OM3 OM4 Unit

PMD Connection PMDConnectionPatchCord

PatchCord

Cable

Description Type OM3 OM4 Unit

Operating Distance Max 100 150 m

Cabling Skew Max 79 ns

Cabling Skew Variation Max 2.5 ns

Channel Insertion Loss Min 0 dB

Channel Insertion Loss Max 1.9 1.5 dB

The total connection loss allocated for OM4 MMF is 1 dB maximum (or 0.5 dB per connector)

17

RELATIVE ECONOMICSRELATIVE ECONOMICS

Equipment Pricing ScenarioEquipment Pricing Scenario• Some equipment examples:

Layer 3 Switch - 52 Slot (48 x 10G + 4 x 40G)Switching Throughput: 1.28 Tbps

QSFP+40GBASE-SR

Module

SFP+10GBASE-SR

Module

10G SRServer Adapter

2 ports

Relative Economics

Equipment (both ends) $/port/10GbE

Relative Economics

$3,500.00

Equipment (both ends) $/port/10GbE

$2,000.00$2,500.00$3,000.00

$500.00$1,000.00$1,500.00

$0.00$500.00

MMF 10G Sw-Sw MMF 10G Sw-Svr MMF 40G Sw-Sw

Vendor A Vendor BVendor A Vendor B

MIGRATION FROM 10G TOMIGRATION FROM 10G TO 40G/100G ETHERNET

Pre-terminated CablingPre terminated Cabling

• The use of pre-terminated optical fiber cabling canThe use of pre terminated optical fiber cabling can facilitate the migration path from 10G Ethernet to 40G and 100G Ethernet for– Switch-to-Server connectivity– Switch-to-Switch connectivity

Migration from 10GBASE-SR to 40GBASE-SR4

2 x 12-fiber MPO connectors

Cabling for 10GBASE-SR

1 x 24-fiber trunk cable assemblyMPO connectors

2 MPO t2 x MPO to12 x duplex LC

Cassette

Cabling for 40GBASE-SR42 x 12-fiber MPO connectors

6 x 24-fiber trunk cable assemblies

12 x MPO adapter

Optical Loss Budget Calculations

100 meters

Using OM3 or OM4 fiber cables• MPO connectors as specified in TIA 568-C.3 have a loss of up to 0.75 dB per connector allows a maximum of two

0.75 0.75

100 metersconnector allows a maximum of two connectors for a 100 meter channel

• Pre-terminated cable assemblies are il bl th t l l MPO

100 meters

available that use low loss MPO connectors with a maximum loss of 0.35 dB per connector

• The use of low loss connectors allows

Worst case

0.35 0.35 0.35 0.35The use of low loss connectors allows up to four connectors for a 100 meter channel

100 m Link length Worst case connectors Low Loss connectors

No. of Connectors 2 2 3 4Connector Loss (dB) 1.5 0.7 1.05 1.4

Optical Fiber Loss (dB) 0.3 0.3 0.3 0.3Optical Fiber Loss (dB) 0.3 0.3 0.3 0.3Channel Insertion Loss (dB) 1.8 1.0 1.4 1.7

meets 1.9 dB budget

Optical Loss Budget Calculations

• The optical loss calculations for 150 h l h i h

Using OM4 fiber cablesand low loss connectors

meter channels are shown on right

• Up to three, low loss, MPO connectors can be accommodated for a 150 m h l i i d OM4

150 meters

channel using pre-terminated OM4 optical fiber cable assemblies

0.35 0.35 0.35

150 m Link length Low Loss connectors

No. of Connectors 2 3Connector Loss (dB) 0.7 1.05

Optical Fiber Loss (dB) 0.45 0.45Channel Insertion Loss (dB) 1.05 1.5

meets 1.5 dB budget

Optical Fiber Array PolarityOptical Fiber Array Polarity

• TIA 568-C.0 standard outlines sample methods ATIA 568 C.0 standard outlines sample methods A and B for maintaining the polarity of parallel array systems

• The following slides illustrate method A and Method B polarity schemes, as well as a version of Method B polarity scheme using Type A (key up to key down) MPO adapters

40G Optical Fiber Array PolarityMethod A for Parallel SignalsMethod A for Parallel Signals

40G Optical Fiber Array PolarityMethod B for Parallel SignalsMethod B for Parallel Signals

40G Optical Fiber Array PolarityMethod B Using Type A AdaptersMethod B Using Type A Adapters

EXAMPLE IMPLEMENTATIONSFiber Connectivity

EXAMPLE IMPLEMENTATIONS

Example ImplementationExample Implementation Migrating from High Density

LC Duplex ModulesMigrating from Standard Density

LC Duplex Modules LC Duplex Modulesto MPO Adapters

• 18 connectors per unit

LC Duplex Modulesto MPO Adapters

• 12 connectors per unit

10G 10G

40G 40G

50% Increase in Capacity50% Increase in Capacity

Example Implementationh h lHigh Density Patch Panel Design

4 cassettes / 48 LC Duplex ports 4 HD Adapter frames / 72 MPO ports

Example Implementationp p

High density rack mountU t 72 LC D l /MPO P t

1U

High density rack mount fiber connectivity

t

Up to 72 LC Duplex/MPO Ports

2U systemsUp to 144 LC Duplex/MPO Ports

2U

4U

Up to 288 LC Duplex/MPO Ports

MPO Trunk Cable AssembliesMPO Trunk Cable Assemblies

12 fib MPO T k bl bli d t i

Cable construction Type of Connectors

12-fiber MPO Trunk cable assemblies are made to measure using a Product Engineering Code that specifies

Flame ratingFiber typeFiber count

Polish / performancePulling eyeLength in meters

MPO Trunk Cable AssembliesMPO Trunk Cable Assemblies

MPO T k bl bli i lti l f 12 fib d t

Cable constructionFlame rating

Type of ConnectorsPolish / performance

Fan-out Length,Construction

MPO Trunk cable assemblies in multiples of 12 fibers are made to measureusing a Product Engineering Code that specifies

Flame ratingFiber typeFiber count

Polish / performancePulling eyeLength in meters

Construction,and Geometry

OPTICAL LOSS TESTINGIt’s not as simple as it seems

OPTICAL LOSS TESTING

Accurate Loss MeasurementsAccurate Loss Measurements

• Why is it important?Why is it important?– Tight loss budgets for a channel

Measurement uncertainty can quickly use up the– Measurement uncertainty can quickly use up the available the loss budget

Testing Accuracy

Several major factors can affect the testing f ti l l taccuracy for optical loss measurements

1. the type and quality of the “test reference cord”

2. fiber mismatch between the test reference cords and the link under testcords and the link under test

3. the characteristics of light source and how light i l d i h fibis coupled into the fiber

Type and Quality of the “Test Reference Cord”Type and Quality of the Test Reference Cord

• end face geometry can have a significant effect on the test• end face geometry can have a significant effect on the test results and must meet precise parameters such as radius of curvature, apex and fiber protrusion

• the performance can be greatly diminished when the surface of the cord connector is scratched or dirty

– inspect the fiber connector with a fiberscope and clean the connector if it is dirty with a dry cleaning technique

Are Your Optical Loss Measurements Reliable?Are Your Optical Loss Measurements Reliable?

• Connect a “known good patch cord” to your equipment as if it g p y q pwere a fiber link and test it using the same test reference cords that are normally used for link testing

• Compare the measured result in the field to the loss value of the factory tested patch cord

• If the result is significantly different (greater than 0.2 dB), something is wrong

– It can be your equipment, your test method or your test cords

Fiber Mismatch Loss

• Fiber mismatch loss is an intrinsic coupling loss due top g

Core diameter differences

numerical aperture differencesnumerical aperture differences

core and cladding concentricity

• Fiber mismatch loss can be quite significant when connecting two fibers at the minimum and maximum of tolerance limits

• Fortunately, most fibers today are manufactured to tighter tolerances than worst case specifications

Launch Conditions

Launch conditions can have the greatest effect on optical l t f lti d fib

• different distributions of launch

loss measurements of multimode fiber

power (launch conditions) can result in different loss measurements

th l h t i d l

launchspot size and

angular • the launch spot size and angular distribution is controlled through

an optical system that couples the light into the fiber

angulardistribution

What is Encircled Flux?What is Encircled Flux?• Encircled flux (EF) is a new method of controlling the power

distribution of light that is launched from the end of a testdistribution of light that is launched from the end of a test reference cord (TRC)

• EF was developed by optical experts to reduce variability in li k l t d t t diti l th d th tlink-loss measurements compared to traditional methods that use an overfilled launch (OFL) source and a mandrel wrap

• The Encircled Flux test method is specified in TIA TIA-526-14-pB, “Optical Power Loss Measurements of Installed Multimode Fiber Cable”

• Encircled Flux is also the test method that is specified in the• Encircled Flux is also the test method that is specified in the IEEE 802.3ba Standard for 40 Gb/s Ethernet (40GBASE-SR4) and 100 Gb/s Ethernet (100GBASE-SR10).

EF Template for 50 μm Fiber at 850 nm

Reference: TIA-526-14-B, October 2010 Source: TIA-526-14-B

What is the Difference in Optical Loss Measurements Using OFL versus EF ?Measurements Using OFL versus EF ?

• the optical loss of three different 20 meter, 2-connector, permanent links p , , pwas measured using two different OLTS instruments

• the same test reference cords (using regular MMF) and mandrels were used when testing the permanent links for both instrumentswhen testing the permanent links for both instruments

• the test setup is shown below:

LC LC

20 m

LC LC

Receive cordi – 22mm Mandrel a) PL MMF

b) PL BIMMF Vendor_Ac) PL BIMMF Vendor_B

OLTS #1

i 22mm Mandrelii - EF Module

OLTS #1OLTS #2

Insertion Loss Measurements

Insertion Loss on 20m Permanent Link with two LC connectors

0 200

-0.100

0.000

OLTS #1 TRC #1

22mm mandrel

OLTS #1 TRC # 2

22mm mandrel

OLTS #2 TRC #1

22mm mandrel

OLTS #2 TRC # 2

22mm mandrelOLTS #2 EF module #1

OLTS #2 EF module #2

OLTS #1 EF module #1

OLTS #1 EF module #2

-0.500

-0.400

-0.300

-0.200

-0.900

-0.800

-0.700

-0.600

IL (d

B)

-1.300

-1.200

-1.100

-1.000

OFL OFL EFL EFL

0.5 dB

-1.400

Test ConditionsVendor A BI-MMF (PL 20m) Regular MMF (PL 20m) Vendor B BI-MMF (PL 20m)

OFL OFL EFL EFL

Optical Loss Testing SummaryOptical Loss Testing Summary• Optical loss testing in the field is not as simple as it

seems and can be affected by many variables includingseems and can be affected by many variables, including type and quality of the test reference cords, fiber mismatch and launch conditions (OFL vs. EFL)

• The Encircled Flux test method is recommended by industry experts for accurate optical loss measurements for both regular MMF and BI MMFsfor both regular MMF and BI-MMFs

• Encircled Flux is also the test method specified for 40G and 100G Ethernet applicationsand 100G Ethernet applications

Something NewgNew Addendum to TIA 942-A “Cabling Guidelines for Data Center Fabrics”

Example of Data Center Fabric Fat-Tree ArchitectureAll access switches are connected to every interconnection switchRequires more switch-to-switch cabling than traditional architectureq g

The Road Ahead for FiberThe Road Ahead for Fiber

• There is a growing need for high speed switch-to-switch and g g g pswitch-to- server connections– Relative economics ($/port/10 Gbps) is favorable for

40GBASE SR d ith 10GBASE SR40GBASE-SR compared with 10GBASE-SR• The use of 12-fiber pre-terminated cabling with MPO

connectivity provides a seamless migration path from 10G to y p g p40G in the future

• Migration from 10G to 100G is also easily accomplished using 12 fib bl bli d d i ll i ltwo 12-fiber cable assemblies today and potentially using only

one 12-fiber cable assembly in the future (new 100G IEEE 802.3 study group)