Towards the FAIR Timing System using White Rabbit s

17 January 2012 Dietrich Beck, EE(BEL)

Towards the FAIR Timing System using Towards the FAIR Timing System using White RabbitWhite Rabbitss

• Towards FAIR• White Rabbit• Timing System


AcknowledgementsAcknowledgements

• GSI Timing Team: Marcus Zweig, Stefan Rauch, Mathias Kreider, Cesar Prados, Wesley Terpstra, Ralph Bär, Dietrich Beck former members: Tibor Fleck, Sergio Mauro

• CERN Timing Team: Tomasz Włostowski, Javier Serrano, Maciej Lipinski, Evangelia Gousiou, Erik van der Bij, Jean-Claude Bau, Pablo Alvarez

• GSI/EE: Jan Hoffmann, Nikolaus Kurz, Holger Brand, …


Towards FAIR…Towards FAIR…

17 January 2012 Dietrich Beck, EE(BEL) 23 November 2011

17 January 2012 Dietrich Beck, EE(BEL) 8 December 2011


Bild von heuteBild von heute

12 January 2012

17 January 2012 Dietrich Beck, EE(BEL) 12 January 2012


General Machine Timing System @ FAIRGeneral Machine Timing System @ FAIR

• parallel execution of beam production chains

• cycles: 20ms to hours

• trigger and sync. equipment actions

• 1 µs precision in 99% of all cases

• few ns precision for kickers

• (ps for rf-systems: BuTiS)

• many rings

• > 2000 devices connected to timing system

• large distances

• robustness: lose at most one message per year


Idea: Timing System Based on TimeIdea: Timing System Based on Time

Not:

1km distance: 5s propagation time

1s precision: requires compensation for cable length

Timing Master Equipment 1do this

Equipment 2do this


Idea: Timing System Based on TimeIdea: Timing System Based on Time

Instead:

• equipment pre-programmed for autonomous action at a given time• pre-programming via timing-events• required: clock synchronization ns required distribution of information and execution of action are decoupled !!!distribution of information and execution of action are decoupled !!!• timing-events must be sent “early enough”timing-events must be sent “early enough”

• upper bound latency for transmission (e.g. 100upper bound latency for transmission (e.g. 100 s): real-time!s): real-time!

• lossless transmission: robustness!lossless transmission: robustness!

Timing Master Equipment 1

do this @ 15:03

Equipment 2

do this @ 15:03 do that

@ 15:01

"Oh dear! Oh dear! I shall be too late!"

???


White Rabbit Cooking Recipe I: White Rabbit Cooking Recipe I: ClocksClocks

• network: Gigabit-Ethernet

• PTP (Precision Time Protocol) IEEE1588-2008

with value t1.. t4 one can• calculate one-way link delay

• syntonize clock rate by tracking the value

• calculate clock offset


White Rabbit Cooking Recipe I: White Rabbit Cooking Recipe I: ClocksClocks

• network: Gigabit-Ethernet

• PTP (Precision Time Protocol) IEEE1588-2008– free-running oscillators on network nodes– need to re-sync often: lot’s of traffic, bad for determinism– only 1-100 s synchronization of clocks

• SyncE (Synchronous Ethernet)– receiver’s clock recovered from 125MHz carrier– 8ns precision

• precise phase measurement and adjustment

• clock synchronization with sub-ns precision and low-ps jitter


White Rabbit Cooking Recipe II: White Rabbit Cooking Recipe II: NetworkNetwork

System TimingMaster

multi-layeredWR Switches WR

NodesGPS• Clock propagation:

• Determinism– “timing events shall arrive early enough”– upper bound latency from timing master to nodes– no handshake: use UDP or raw Ethernet (no TCP)– switches implement cut-through for high-priority messages

• Robustness– bit errors: Hamming code– packet loss: Reed-Solomon code

• Redundancy– redundant links between switches– link aggregation (in combination with Reed-Solomon)


transceiver

FPGA

VCO 20MHz

VCO 125MHz

WR clock!!!

WR Cooking Recipe III: Implementation…WR Cooking Recipe III: Implementation…

• path asymmetry (two cables) impacts phase compensation White Rabbit PTP

– one bidirectional single-mode optical fiber– different wave lengths for TX/RX: speed differs slightly (dispersion)– link-delay-model

• relative delay coefficient α depends only on the type of fiber (off-line)

• fixed delays are measured on-line

• White Rabbit node

host bus interface


Inside the FPGA:Inside the FPGA:

within FPGA: Wishbone “System-on-Chip” bus for interconnection of IP cores

FPGA


Transmission of Transmission of Timing-EventsTiming-Events: Etherbone: Etherbone

• idea: Connect distant Wishbone buses via a network protocol layer. Ethernet + Wishbone = Etherbone

• direct memory access to devices via White Rabbit network

• used for timing-events


The White Rabbit ProjectThe White Rabbit Project

• development in the frame of CERN’s renovation and the FAIR projects

• collaboration: CERN, GSI, …, industry partners

• Open HardWare Repository: www.ohwr.org– Open Source: Software and HDL– “CERN Open Hardware License”: Hardware

• commercial production and support

“The White Rabbit (WR) project is a multi-laboratory and multi-company effort to bring together the best of the data transfer and timing worlds …” (Proc. ICALEPCS 2009)

http://www.ohwr.org/


Timing Systems @ FAIRTiming Systems @ FAIR

General Machine Timing System• based on White Rabbit• fairly cheap: € 600,- SPEC board• sub-ns synchronization• distribution of

• (clock)• time-stamps• timing-events

• BEL group

• one global timing master• clock derived from BuTiS

Talk T. Fleck, FAIR Technikforum

Bunch phase Timing System (BuTiS)

• invest € 15k per receiver station• ps precision (100ps/km accuracy)• distribution of

• clocks!!!• NO time-stamps at FAIR• NO timing-events

• HF group

• one global BuTiS center

Talk P. Moritz, FAIR Technikforum

Remark: If you are interested time-stamps, you need White Rabbit.

If you are interested in the most accurate clocks, you need BuTiS.

If you are interested in both, you need both.


1. settings management (via normal network)a. preloads set-values to the FECs (set-value can also be a ramp…)b. provides schedule to timing-master

2. according to schedule, timing-master broadcasts timing-events to FECs (timing-events: time of execution and reference to a set-value)

3. timing-receiver of FEC: timely triggering of FEC, FEC applies set-value

The FAIR Control System (slightly simplified The FAIR Control System (slightly simplified ))

Timing

Timing MasterFront-end Controller

Front-end Controller

Settings Management

1b: Data supply:schedules, alternatives

1a: Data supply:set-values

3. timing receiver

2: broadcast timing-events


The FAIR Control System (slightly simplified The FAIR Control System (slightly simplified ))

Timing

Timing MasterFront-end Controller

Front-end Controller

Settings Management

Data supply:schedules, alternatives

Data supply:set values

timing receiver

Analogy to performing a classical piece of music:

• composer: settings management

• conductor: timing master

• musicians: FECs

• move of conductor's baton: timing-events - one timing-event may trigger multiple FECs


Timing Master

Control System (application layer)

LSA (settings management)

AP

I

Equipment 1 Equipment N

FE

SA

WR

Equipment 2 Equipment 3

Beam Interlock SystemPost Mortem System

AP

I

BuTiSanother link

Equipment 4

a link

Equipment ..

CommonSystems

Alarm, Archive, Logging, Oscilloscope

AP

I

BuT

iS

Disclaimer: This is my personal view and may be incorrect and not up-to-date


BuTiS and WRBuTiS and WR


What the Timing System is AboutWhat the Timing System is About

• campus wide distribution system for– timing-events– time-stamps– other machine or beam related telegrams

• not limited to accelerator controls

• available to “everyone”

• typically, data is propagated top-down

• other traffic only in exceptional cases– kicker control– bunch-to-bucket transfer– beam interlock notification


What the Timing System is NOT AboutWhat the Timing System is NOT About

• general purpose communication

• transmission of bulk data

• personnel protection system

• things not related to machine timing

Exceptions only in well justified cases

• FECs not connected to FESA

• Info-Service

• part of machine protection system


Next Steps…Next Steps…

@GSI: Set-up a first network 2012

• usage in a field environment• investigation of stability and long-term effects• connection between new and existing timing system• first tests with existing machines at GSI


……Next StepsNext Steps

@GSI/CERN: WR starter kit (timing system demo) spring 2012

@Saclay: Low energy part of p-linac: first WR based timing system provided by GSI (early 2013)

@CERN: • Installation of WR based timing system at AD ring (2013) • Installation of WR link between CERN and Gran Sasso (2012)


StatusStatus

• Slovenia: In-Kind contribution timing receivers. Receivers and FPGA building blocks for FAIR timing system specified.

• Germany/GSI: In-Kind contribution General Machine Timing System: specification until March 2012.

• White Rabbit PTP link between two nodes established• sub-ns synchronization with a jitter of hundred ps• propagation of WR PTP over three layers of switches• FPGA building blocks for simple timing system exists• integration work in progress• planning of WR network cabling in progress

• First WR nodes developed by EE


Further readingFurther reading

• www.ohwr.org

• https://www-acc.gsi.de/wiki/Timing/TimingSystemDocumentation (search for “FAIR timing system”)

http://www.ohwr.org/

https://www-acc.gsi.de/wiki/Timing/TimingSystemDocumentation

https://www-acc.gsi.de/wiki/Timing/TimingSystemDocumentation


Moersbacher grundMoersbacher grund

view towards SIS 100, 23 November 2011

Towards the FAIR Timing System using White Rabbit s

Documents

Transcript of Towards the FAIR Timing System using White Rabbit s