White Rabbit Sub-Nanosecond timing over Ethernet

H.Z. PeekNikhefAmsterdamElectronics- Technology

VLVnT11Erlangen12-14, October 2011 1

White RabbitWhite RabbitSub-Nanosecond timing over Sub-Nanosecond timing over

EthernetEthernet

H.Z. Peek ...H.Z. Peek ...... on behalf of the White Rabbit ... on behalf of the White Rabbit collaborationcollaboration

1) Alice in Wonderland, Lewis Carroll (1865)

“Oh dear! Oh dear!I shall be too late!” 1)


VLVnT11Erlangen12-14, October 2011

Outline

2

What is White RabbitNetworkBuilding blocksMeasurement resultsApplicationsConclusion



What is White RabbitNetworking and timingBased on well-known technologies /

standardsOpen Hardware and Open SoftwareInternational collaborationMain features:

Transparent, high-accuracy time distribution,

Low-latency, deterministic data delivery,

Designed for high reliability,Plug & play.



Timing over EthernetTiming over EthernetHistory

1985NTP (RFC 1305) sub milli-second

2002

PTP (IEEE 1588) sub micro-second2012

White Rabbit (?) sub nano-second



White Rabbit timing distribution

Synchronization with sub-ns accuracy over fiber

A combination of:Synchronous Ethernet (SyncE)

syntonizationDigital Dual-Mixer Time Difference

(DDMTD) phase detection (Precision Timing Protocol (PTP)

synchronization



Timing distribution

Averaging all clock edges• “mean phase”

Syntonization• “The adjustment of two electronic circuits or

devices in terms of frequency”

ReferenceClock

Master

SFP

SlaveRx

SFP

Tx

TxRx

SFP

SlaveRx

SFP

Tx

TxRx

High PrecisionLow jitterHigh Precision

Low jitter

Ordinary serial data

communication channels

Ordinary serial data

communication channels!



toffse

t

7

RefClk

Cnt

Master

SFP

SlaveRx

Tx

SFP

Tx

Rx

PTP (IEEE1588)

Cnt

t1t2 (= Cnt + SlaveBitSlide)

Sync message

Follow_Up Message (t1)t1 t2

t3 t1 t2Delay_Req Message

Delay_Response Message (t4)t4 t3 t1 t2

t1 t2

MasterClockTime

SlaveClockTime

t3

t4

t4(= Cnt + MasterBitSlide + )

=0

Data (Packets)Clock

Time Stamp

t2

t3t4

t4

t1

t1

1. t2 –t1=offset + MSdelay

2. t4 –t3=-offset + SMdelay

3. MSdelay=SMdelay



White Rabbit network



White Rabbit Switch V2

Central element of White Rabbit network

Fully custom design, designed from scratch

10 SFP ports (1000Base-LX) Capable of driving long distance Single

Mode fiber



White Rabbit Node

http://www.ohwr.org/projects/spec/wiki

Simple PCIe FMC Carrier (SPEC)

Currently available



White Rabbit Switch V3

2 uplink ports, 16 downlink portsHardware just assembled.Hardware & Software currently being

tested.Production expected Q1-2012http://www.ohwr.org/attachments/741/White_Rabbit_Technical_Spec_05.pdf



Measurement test setup

http://www.youtube.com/watch?v=ZSRQEExbdq8

Hot-air gun demo:



Measurement results



Digital to Time Converter (DTC) = Alarm Clock

Time to Digital Converter (TDC) = Time stamping

14

Applications



White Rabbit enables measurement and control applications which are using distributed system technologies.

Such applications may be spread over large distances.

Data transmission delay changes are continuously measured and compensated.

System timing of White Rabbit nodes are synchronized with high precision.

15

Conclusions



White Rabbita multi-laboratory, multi-multi-laboratory, multi-

company effort company effort

Thank youComing Soon!



How do we know the time e offset between Master and offset between Master and

Slave?Slave?

Clock & Data coded into one stream DC-Balance Special code-groups / Word Alignment

1

2

3

Toffset = Total delay(Master>Slave>Master) / 2Can we measure propagation delay

using existing serial communication channels?Serial Communication Coding

Properties:



Measure propagation delay using FPGA SerDes

technology

Start

LatticeLFSCM25

SerDes

Lattice SCPCI Expressx1

Evaluation Board

XilinxVirtex-5

SerDes

XilinxML507 Board

LEDs

Stop

100 KmFiber

19

Transmitter

SFP

Receiver

SFP

1

3

Regenerate system clock at the receiverUsing a barrel shifter for word alignment

delay known with bit clock resolution

System

Clock

x 20SystemClock



Master

Slave

Start

Stop

10 Km

fiber

Stop

Clock Loopba

ck(DPLL)

VCXO

DAC

Time offset measurement test setup

Loopback the recovered clock with a Voltage Controled Xtal Oscillator



0 0001 = 1 0 0000 = 0 0 0011 = 3

Resynchronization + Barrel shifting action

RxRecClk

BitSlide(4:0)

011101011000001010110111010110000010101101110101

0

Algorithm:Propagation Delay = “Start-Stop” Delay + “LED Value” * UI

Start/Stopdelay

3 1

Unit Interval

(UI)

Details : VLVnT09, October 15, 2009 in Athens• Presentation: http://www.nestor.noa.gr/vlvnt09/pres/Jansweijer_MeauringPropagationDelay.ppt

• Paper: http://dx.doi.org/10.1016/j.nima.2010.04.126



ReferenceClk

Fine timeCount

Coarse time

Sta

rt

Sto

p

Master

SFP

SlaveRx

Tx

SFP

RxUsrClk

TxUsrClk

TxUsrClk

RxUsrClk

1. # of system clocks

2. # bit clocks (i.e. barrel shifts)

3. Phase between Master node Tx and Rx clock

toffset

Tx

Rx

Measure time offsetBidirectional + Loopback the

recovered clock1.25 Gbps (IEEE802.3 1000BASE-X = Gigabit

Ethernet)

Time offset is determined by:

For details please see Technical Report “ETR2010-01”:

http://www.nikhef.nl/pub/services/biblio/technicalreports/ETR2010-01.pdf

1.25 Gpbs

10 .. 100 ps

20 x #bit clocks

800 ps



Master time

tpd1

tpd1

Slave time

toffset

tdisp

mTx sRx

mRx sTx

1

2

Time offset and fibre dispersion

000289.210.9.4

14162.2

61

2

1

PP

dDV

Ptoffset

(tdisp = 15150 ps over 10.7 km => D() = l416 ps/km)

White Rabbit Sub-Nanosecond timing over Ethernet

Documents

Transcript of White Rabbit Sub-Nanosecond timing over Ethernet