100 Gb/s: The High Speed Connectivity Race is On

Cathy Liu

SerDes Architect, LSI Corporation

Harold Gomard

SerDes Product Manager, LSI Corporation

October 6, 2010

2

Agenda

• 100 Gb/s Ethernet evolution

• SoC challenges and solutions to drive 4x25 Gb/s Ethernet

• High Speed Serial link system design

• Summary

3

Contemporary 100 Gb/s Ethernet with 10 Gb/s SerDes

100 Gb/s Media Access Controller (MAC)

10 x 10 Gb/s SerDes

100 Gb/s Physical Coding Sublayer (PCS)

Optional Forward Error Correction (FEC)

20 “virtual” lanes

10 “physical” lanes

2:1 bit multiplexer

10x10 GbE

Se

rDe

s

Se

rDe

s

Today’s 100 Gb/s Ethernet built from 10 lanes of 10 Gb/s SerDes

Challenges for 10 x 10 Gb/s implementation implies high cost

Uses many package and/or connector pins

Complicates board routing and more high speed signal layers

Limits ASIC and faceplate port densities

Large number of pairs cabling to terminate implies higher cost

Large outer diameter implies heavy, inflexible cable

20 differential pairs per interface

4

Industry Trend for High Speed Serial Link System

• Merging 25-28 Gb/s serial electrical links

– IEEE 100GbE, OIF CEI 28G-SR/25G-LR, 32GFC, Infiniband EDR

• Lower-cost, more manageable cable assemblies

• Common form-factors and connector technologies

4 x 25 Gb/s is the natural evolutionary step for 100 Gb/s Ethernet

10x10 GbE

Se

rDes

Se

rDes

4x25 GbE

Se

rDes

Se

rDes

5

Path from 10x10 Gb/s to 4x25 Gb/s: Challenges

Challenges

Lossy links and noisy environment

Need High speed circuit design at 25-28 Gb/s

Technology and implementation limitations

Large channel counts (~200) in SoCs 4x25 GbE

SerD

es

SerD

es

BladeSystem

10x10 GbE

SerD

es

SerD

es

- 32dB IL at 5GHz

- 30dB Xtalk

Legacy Channels already present significant challenges @ 10 Gb/s !

6

Path from 10x10 Gb/s to 4x25 Gb/s: Solutions

4x25 GbE

SerD

es

SerD

es

10x10 GbE

SerD

es

SerD

es

Addressed Challenge Solution Space

High-Loss Channel Alternative Signaling Format Beyond NRZ

High-Loss and Noisy Environment ADC + DSP-based SerDes Architecture

Noisy Environment FEC and Noise Cancellation

Technology and Circuit Limitations Innovative Circuit Implementation

Large Channel Counts Advanced SoC Integration

All Joint System-Design Optimization

7

SerDes Architecture Evolution

• Industry trends are to consider alternative signaling formats beyond NRZ like

Multi-level or Multi-band for 25G+ long reach application

• Low power and high speed interleaved ADC technology enables DSP

architecture to extend equalization capability

• Forward Error Correction (FEC) and crosstalk cancellation to further improve

SNR

ADC 1

ADC 2

ADC n

Interleaved

ADC

FF

E

Analog Filter

AGC

DF

E

FE

C

Sys

tem

In

terf

ac

e

Digital Domain

Timing

Recovery

Analog Front End

Reference: Liu et al, “Comparison of signaling and equalization schemes in high speed SerDes (10-25Gbps)” DesignCon 2007

8

Implementation Challenges for Building a 25G SerDes

Circuit innovations required to tackle 25G SerDes challenges

Challenge Requirements Constraint

Signal Integrity RL of 8dB @ 12.5GHz Low power

Tx Driver Bandwidth 18.75GHz Low power

Signal Dynamic Range

Scaling30dB Linearity

28nm CMOS device

headroom @ 0.85V

Ultra-low PLL Random Jitter 250fs (rms) Low power

Reference: Wu et al, “A 2x25Gb/s Deserializer with 2:5 DMUX for 100Gb/s Ethernet applications” ISSCC 2010

9

Circuit Innovation to Tackle Implementation Challenges (I)

• Source-Series Terminated (SST) driver with a bridge inductor

– CMOS differential circuit provides Ultra-high bandwidth for 25 Gbps signal processing

– Bridge inductor mitigates return loss degradation from I/O capacitance

SST driver simulations on LSI 16G-LR SerDes meet 8dB RL spec @ 25 Gbps

TX Return Loss Simulation of 16G-LR driverSST Driver with Inductor

Blue: without inductor

Black: with inductor and power mesh

Pink: with inductor

With Inductor

Without Inductor 8dB RL spec @ 25 Gbps

10

Circuit Innovation to Tackle Implementation Challenges (II)

• ADC-based SerDes for 25Gbps Serial link with alternative signaling

– 12.5Gs/s 6-bit low-power time-interleaved ADC enables DSP for 25G SerDes

– DSP provides powerful equalization to tackle ISI, crosstalk and reflection

ISI Correction

CDR

Gain Adaptation

Skew Adjustment

ADC Calibration

Data

Ali

gn

me

nt

Dec

od

ing

Bu

bb

le C

orr

ec

tio

n

11

High Channel Count SerDes Integration in ASIC

Challenges in high channel count

SerDes integration• High-speed Clocking within SoC

• Power Integrity of SoC

• High-speed Signal Integrity

Asymmetrical TX and RX

12

Challenges in high channel count SerDes integration- PLL Coupling in High Channel Count SerDes Integration

• LC-PLL Coupling is one of top challenges for high-speed clocking

• Advanced techniques are required for multiple LC-PLL placement– Minimum spacing between 2 LC-PLLs

– Design rule for substrate isolation and power mesh

Advanced LC-PLL placement achieves no injection lock coupling from other PLL

PLLB

Test Condition

PLLA @ 14.025G

p2p Jitter

8.5G 12.3ps

10.3125G 13.2ps

12.5G 12.9ps

14.025G 11.2ps

Powered off 12.6ps

PLLA Clock Out

PLLA = 14.025 GHz, PLLB = 8.5 GHz

-80

-70

-60

-50

-40

-30

-20

-10

0

1.600

E+09

2.000

E+09

2.400

E+09

2.800

E+09

3.200

E+09

3.600

E+09

4.000

E+09

4.400

E+09

4.800

E+09

5.200

E+09

5.600

E+09

PLLA Clock Out

Measured Jitter of PLLA

Variation limited to +/- 1psSpectrum of LSI 40nm 14G-LR SerDesPLLA Clock Out

PLLA=14.025 GHz, PLLB = 8.5 GHz

13

Challenges in high channel count SerDes integration

- Power Integrity in High Channel Count SerDes Integration

Power supply

Recommended

PKG DCAP PKG IR Drop (%)

Self-induced on-die

AC noise peak voltage (%)

CAP Value Spec Simulated Spec Simulated

VDDA 22nF (x2) 1.0 0.76 1.25 0.81

VDDREF 10nF (x2) 1.0 0.23 1.25 1.02

VDDA1P8 N/A 1.0 <0.5 1.25 0.5

Simulation Setup Simulated AC Noise

PI Analysis of SoC with LSI 14G-LR SerDes

PI analysis ensures IR drop and AC noise meet SoC design requirements

14

Challenges in high channel count SerDes integration - Package SI for High Channel Count SerDes Integration

• Package routing length can vary 12mm to 25mm for high channel count SoC

TX insertion loss TX return loss RX-TX cross talk

Package SI Kit provides powerful tool for High-Speed SI analysis

Package SI Analysis of SoC with LSI 14G-LR SerDes

0 1 2 3 4 5 6 7 8 9 10-20

-15

-10

-5

0

Frequency (GHz)

Magnitude (

dB

)

CW108040 Tx Device Sdd22

Return Loss: Worst Case Die over PVT

Port 0: Tx

Port 1: Tx

Port 2: Tx

Port 9: Tx

16GFC EpsilonT Sdd22 RL Compliance Spectrum

15

• Increased loss over given medium length

• System sizes do not shrink

• Fixed budget loss driven by a transceiverElectrical

• Increased xtalk at connector and package

• New BGA routing strategy required

• Port density vs. signal integrity Mechanical

• Dynamic power consumption increase

• Single hot spot ASIC thermal management

• Cannot solve with innovative circuits aloneThermal

Challenges in high channel count SerDes integration - Multiple dimensions

16

High-Speed System Design Considerations: SerDes

• SerDes targets compensation for non-ideal system characteristics

– DJ caused by frequency dependent loss and reflections

– Limited ability to address RJ from excessive crosstalk

LSI Proprietary

SerDes Features System Benefit

TX FIR de-emphasis Reduction in dispersion/loss

RX Linear EQ Reduction in dispersion/loss

RX Floating Tap DFE Compensates post-cursor ISI and reflections

RX DFE loops Improves immunity to Crosstalk

SerDes mitigates the need for advanced PCB material and

Interconnect Solutions…until now

RX Floating Tap DFE

kaka

Tx FIR RX Linear EQ

Transmitter Channel Receiver

+

17

• 10Gbps System Example with severe Crosstalk

High Speed System Design Considerations: Crosstalk

Linear EQ Only

7.5” Megtron-6 40” Megtron-6 7.5” Megtron-6

Linear EQ + DFE

Without Crosstalk With Crosstalk

Joint System Optimization required for crosstalk mitigation

TX Xcede

Connector

Xcede

ConnectorRX

18

High Speed System Design Considerations: Reflection

• 10Gbps System Example with severe Reflections

Mezz Connector

Mezz Card

SoC

SoC

Blade Card

Significant reflections at 5 GHz Nyquist Frequency

0 2 4 6 8 10 12 14 16 18 20-40

-35

-30

-25

-20

-15

-10

-5

0

Frequency (GHz)

Magnitude (

dB

)

Worst Case 5 inch Blade Channel with FCI GigArray Mezz Connector

Sdd21

Insertion Loss: Blade Channel

Sdd11

Return Loss: Blade Channel

Sdd21

Insertion Loss: Ideal 5-inch FR4 trace

5 inch Chip-to-Chip Blade Channel

19

High Speed System Design Considerations: Reflection

• 10Gbps System Example with severe Reflections

LSI Proprietary

1-tap Rx DFEFixed and Floating Rx DFE taps

reflections

5 inch Blade Channel 5 inch Ideal FR4 Trace Channel

Joint System Optimization required for reflection mitigation

20

• High density system applications require truncated search criteria

• Bound the problem by simulating conservative worst case scenarios

• Parameterize problem and limit permutations to only viable options

WC

Scenario SI simulation stress parameter SerDes Rx Stress Goal

1 Maximize IL and XTLK magnitudes Degrade SNR

2Maximize reflections using

short T-lines and/or long stubs

Increase ISI from

reflections

Stressing System simulation model with worst case SI scenarios

High Speed System Design Considerations: WC Scenario

21

25 Gb/s System Design Challenges

• Higher data rates lead to electrically longer channels

• Consequences: diminishing SNR’s and generally higher return loss!

Primary Impairments are More Accentuated at Higher Frequencies

0 2 4 6 8 10 12 14 16 18 20-80

-70

-60

-50

-40

-30

-20

-10

0

Frequency (Hz)

Magnitude (

dB

)

Insertion Loss

Total XTLK

10 Gb/s 25 Gb/s

0 2 4 6 8 10 12 14 16 18 20-20

-18

-16

-14

-12

-10

-8

-6

-4

-2

0

Frequency (Hz)

Magnitude (

dB

)

Return Loss

Less

SNR Higher Return Loss

Reference: Kollipara et al, “Practical Design Considerations for 10 to 25 Gbps Copper Backplane Serial Links,” DesignCon 2006

22

Issue Potential Solutions

Elevated Insertion LossUltra Low Loss Tangent PCB laminates

Ultra Low Profile Copper

Aggravated dispersion Lower Dielectric Constant

Elevated IO Return Loss Innovative circuit / inductors

Elevated Connector Return LossOptimized Mating Interface Impedance

Smaller Compliant Pins

Elevated Connector CrosstalkImproved Shielding

Optimized Footprint Topology

25 Gb/s System Design Issues

Joint System Optimization Required for 4x25 Gb/s

23

25 Gb/s Applications Power vs. System Margins Tradeoff

No SkewSystem Margin Focus

….at manageable power

Low margins

LSI margins

• LSI IP and System design focus on application margins

– 3-sigma design and beyond for some key parameter (latch offset)

– Analog and CML circuits for industry leading performance and margins

– Low Power Design but not at the expense of system margins

24

Summary

• 4 x 25 Gb/s is the natural evolutionary step for 100 Gb/s Ethernet

• SerDes architecture and circuit innovation required for 4 X 25 Gb/s

• SoCs require high quality clocking, Power Integrity and SI analysis

• 100G is a multi-dimensional problem that needs to be tackled holistically

• SerDes alone not a panacea for high speed system design challenges

• Joint System Optimization required for crosstalk and reflection mitigation

25 LSI