Presenter

Krzysztof Horszczaruk

IP Network Expert

Network Development Department

Polska Telefonia Cyfrowa S.A.

1

100G Ethernet Czy znowu mnożymy przez 10 ?

2

Presentation Subject

Piece of History

100G status

Test outcome

3

Past PLNOG 100G related speeches

PCS coding, DWDM transmission, 100G IEEE standards, CFP internals (2010); no slides

http://data.proidea.org.pl/plnog/4edycja/materialy/video/d109a.mpg

FORCE10 Product, CFP evolution, global traffic demand (2011)

http://www.data.proidea.org.pl/plnog/6edycja/materialy/prezentacje/Andreas_Falkner.pdf

CFP developments and evolution (2012)

http://www.data.proidea.org.pl/plnog/8edycja/materialy/prezentacje/Jason_Kleeh_100_GbE_and_Beyond.pdf

100G EKINOPS Optical Solutions (2012)

http://www.data.proidea.org.pl/plnog/8edycja/materialy/prezentacje/Norbert_Gulczynski_100G_in_the.pdf

IPoDWDM Orange Case Study (2012)

http://www.data.proidea.org.pl/plnog/9edycja/materialy/prezentacje/mazepakrzysztof1_100G.pdf

4

ether-story

http://en.wikipedia.org/wiki/IEEE_802.3

• 2,94Mbit/s Experimental 1973 • DIX Ethernet II 1982 • 802.3 10Base5 1983 • 802.3a 10Base2 1985 • 802.3u Fast Ethernet 1995 • 802.3z 1G Ethernet 1998 • 802.3 ae 10G Ethernet 2003 • 802.3 ba 100G Ethernet 2010

5

40th anniversary

http://americanhistory.si.edu/collections/search/object/nmah_687626

National Musem of American History 1973, Robert Metcalf, Xerox Corporation, California, Palo Alto

6

May 22, 1973 That is the first time Ethernet appears as a word, as does the idea of using coax as ether, where the participating stations, like in AlohaNet or ARPAnet, would inject their packets of data, they'd travel around at megabits per second, there would be collisions, and retransmissions, and back-off

Ethernet Birthday

http://www.wired.com/wired/archive//6.11/metcalfe_pr.html

David Boggs Robert Metcalfe

November 11, 1973 the first day the system actually functioned

7

30th anniversary

http://tech.mattmillman.com/10base5/ 8

decrypt

9

100GBASE-SR10

• One 24 fiber-cable, MPO connector

• 20 active fibers (10-TX, 10-RX)

• 10 Gb/s per fiber

• One wavelength per fiber

• OM3 or OM4 multimode fiber

• 100 or 150 meter reach

• Expensive fiber – ribbon cables, MPO connectors

• Cheap transceivers – 850 nm VCSEL technology

10

100GBASE-LR4

• One 2-fiber Single Mode FO cable

• 4 wavelengths per fiber (LAN WDM Grid)

• 25Gb/s per wavelength

• 10 km reach

• Cheap Fiber –standard SM FO pairs

• Expensive Transceivers – Gearbox, 1310nm lasers, plus WDM-Mux/Demux

• 100GBASE-ER4 – allmost the same

extended reach: 40km

Better Laser on TX and SOA amplifier on RX

1300nm band

11

LAN WDM grid: real wafe lengths

10km 100Gb/s: 800 GHz spacing LAN WDM

decided by voting: 64% (CWDM grid was an alternative)

Can be same grid as 40km 100GbE (CWDM could not)

wafelength calculated assuming c = 2,99792458 * 108m/s

http://www.finisar.com/sites/default/files/technical-docs/100GbsAndBeyondEthernetOpticalInter.pdf

12

13

64b/66b PCS Goals:

• clock recovery

• transition density

• stream alignment

• DC balance

Rationale:

• 8B/10B has 25% overhead

• 10 Gb/s with 8b/10b coding would require 12.5Gb/s laser

• 12.5 Gbps lasers were not expected to become available for several years

• OC-192 lasers were available, so it has been decided to develop new coding

”Weaknesses”:

• no guarantee for transition density (64 0s every 58 years on 10G)

• no guarantee for DC Balance (is it needed on FO ?) 14

64b/66b for dummies

15

MLD

Packet may be byte-striped across channels

Each channel should have a unique marker to allow skew control at PCS receiver

http://www.ieee802.org/3/az/public/nov08/nicholl_02_1108.pdf

16

CFP

17

CFP-2 / CFP-4

18

QSFP28

19

10G generations

20

100G – where are we now ?

21

some indicators

10x10G motherboard + 10 XFPs :: ~70/30

1x100G motherboard + 1 CFP :: ~50/50

CFP

CFP-2

CFP-4

22

CFP comparison

optic module CFP CFP-2 CFP-4 QSFP28

power avg. 20W 8W 5W ?

pin count 148 104 56 38

electr. TX / RX lenes 12, 11, 10 10, 8, 4 4 4

23

CFP inside

TOSA

ROSA

24

proposed 400G

http://www.finisar.com/sites/default/files/pdf/Beyond%20100G%20Client%20Optics%20-%20Chris%20Cole.pdf

25

ether-lanes

Twisted Pair Cable MM Fiber SM Fiber

10 BASE-T 1

100 BASE-TX 1

1000 BASE-T 4

1000 BASE-TX 1

1000 BASE-SX 1

1000 BASE-LX 1

10G BASE-T 4

10G BASE-LR 1

100G BASE-SR10 10

100G BASE-LR4 4

400G BASE-LR16 1626

Beyond 1T The only approach to control lane count beyond 1T Ethernet is higher order modulation (C.Cole, Finisar)

Multi-Lane (Multi-FO)

Large optical component count PICs tightly coupled with CMOS digital signal processing (DSP) ASICs (C.Cole, Finisar)

To reduce optical component count and move complexity to electronics (to drive down cost) (M.Traverso, Cisco)

27

100G in OTN/DWDM

• ITU defined OTU-4 for 100G transport

• OTU-5 is being discussed (IEEE+ITU-T) http://www.ieee802.org/3/ad_hoc/hse/public/12_09/stassar_hse_01_0912.pdf

• ITU-T considers 400G and 1T over OTN

• They consider multi-lane above 100G

4 x 3808 B 4 x 256 B

1 14 17 3824 3825 4080

28

100G in DWDM

at hi-bitrates, extensive DSP processing is

necessary

DSP chips not yet capable to process

100Gbps serial rate

DP-QPSK is a solution –

quadruple gain

29

STD Mosaic

30

RECAP 100G is ready to use

We are at early stage of 100G

Price drop is expected

Port density will increase

• Multi-Lane transmission (FO or lambdas)

• (Multiple lasers and receivers)

• WDM Mux / DeMux

• GearBox Chip

• MLD sublayer

• Skew control

100G introduced complexity not present

in 10G:

It will get even worse beyond 100G

31

Test Equipment

NE-40E-X8 NE-40E-X16 NE-40E-X8

Spirent Test Center SPT-2U

32

• We tested 100G card, but we have 10G only tester

• „snake” trick has been used to load 100G interface:

L2 pseudowire cascade

L3 VRF cascade

Test setup

33

Test setup

• For Capacity testing additional routers has been used as a rate-converters

34

HA Test Outcome

• NSR test

• OIR Test

3.1 NSR active MPU removal no loss NSR configured

3.1 NSR MPU insert no loss NSR configured

3.1 NSR manual switchover (CLI) no loss NSR configured

3.2 Fabric OIR remove 1st fabric 3,6ms loss

3.2 Fabric OIR remove 2nd fabric 6,1ms loss

3.2 Fabric OIR remove 3rd fabric 2,2 ms loss

3.2 Fabric OIR insert fabric no loss

35

Performance Test Outcome, native IP (linerate packet size)

B

50 B

100 B

150 B

200 B

250 B

Pure Routing uRPF strict uRPF loose + netStream (sampling 10)

+ QoS + L3 ACL + L4 ACL

83 B

126

B

126

B

173

B

173

B

18

9 B

189

B

90 B

116

B

11

3 B

173

B

17

3 B

239

B

239

B IPv4

IPv6

36

Performance Test Outcome, MPLS (linerate packet size)

B

20 B

40 B

60 B

80 B

100 B

120 B

140 B

160 B

Pure MPLS Performance MPLS performance with QoS

137 B 146 B

37

Capacity Test Outcome

• MTU

• LFIB

• RIB-in / FIB

8.1.1 IPv4 MTU 9622 B

8.1.2 Ipv6 MTU 9622 B

8.1.3 MPLS MTU 9654 B

8.2 LFIB Capacity 90 000 LSPs 45 000 ingress + 45 000 transit; actually tester limit; Huawei provided test outcome showing 100 000

RIB-in entries FIB entries

8.3 RIB Capacity 2 GB MPU 6M 2,2M

8.3 RIB Capacity 4 GB MPU (non-standard)

10M 3M

38

Conclusion

Huawei NE-40E routers provide perfomance and capacity for our IP/MPLS Core for several next years

39