PSB RFLL Upgrade

1

Maria-Elena Angoletta, Alfred Blas, John Molendijk CERN BE/RF

Jorge Sanchez Quesada MedAustron

Petri Leinonen Fellow

A LEADING-EDGE HARDWARE FAMILY FOR LOW-LEVEL RF AND DIAGNOSTICS APPLICATIONS IN

CERN’S SYNCHROTRONS

M. E. Angoletta, A. Blas, A. Butterworth, A. Findlay, M. Jaussi, P. Leinonen, T. Levens, J. Molendijk, J. Sanchez Quesada, J. Simonin (CERN)

U. Dorda, S. Kouzue, C. Schmitzer (MedAustron)

Introduction

J.C. Molendijk - LLRF 2013 2

A DSP based digital LLRF developed around 2003 at CERN using a BNL developed motherboard and CERN developed daughter cards and other modules has been successfully used to operate LEIR at CERN since 2005 [1,2,3,4]

A Leading-edge hardware family1 for LLRF and diagnostics2 applications in CERNs synchrotrons has now been developed to succeed this previous (prototype) generation. Similar from a conceptual viewpoint Entirely redesigned using recent components Adheres to recent industry standards [5,7] Extensive use of novel technologies Application of novel on-chip communication architecture [9] Computer to firmware interfaces now defined by a common database [10] Fully remote reconfigurable

1 See poster ID#45 LLRF 2013 «A Leading-edge…» 2 See poster ID#46 LLRF2013 «LLRF & Longitudinal Diagnostics Implementation for CERN’s ELENA Ring»

Outline


Introduction Digital LLRF Principle Hardware Architecture

VXS Standard VXS Switch FMC Standard FMC Modules VXS-DSP-FMC Carrier

Firmware Architecture Wishbone SoC Interconnection Architecture for Portable IP

Cores Memory map management

Project Status Planning Conclusion References


Digital LLRF Principle

Master DDS

SynthRef

Tune

10 MHz

Slave DDS

Ref

Tune

NCO

I/QD

ACRF Clock 67.5 - 125 MHz

Cavity drive0.6 - 1.75 MHz

Base-band I/Q Signal Processing

Frequency Program Hbase Drive I/Q

I/Q

Cavity Return

Radial LoopPhase LoopHbase

B Field

RF Clock 67.5 - 125 MHz

DDC

Ref

Tune

NCO

AD

CI/Q

PSB C02

DDC

Ref

Tune

NCO

AD

CI/Q

DDC

Ref

Tune

NCO

AD

CI/Q

Cavity Return

DDC

Ref

Tune

NCOA

DC

I/Q

Beam Phase Pickup

Radial Pos Pickup Σ

Radial Pos Pickup ∆

Timing


5 J.C. Molendijk - LLRF 2013

The original DSP based Beam Control system [1, 2, 3, 4] Successfully operational in LEIR since 2005

Weak points: Inter DSP communication through panel mounted link-ports DSP memory and VME share the same bus (conflicts operation) Moderate DSP “Fast loop” cycle time 15 μs (DSP Clock 80 MHz + Assembly programming). RF Clock and TAG distribution through external distribution The electronics has become obsolete (DSP FPGAs ~2003) Non standard mezzanine solution with (too) limited on-board FPGA resources


6 J.C. Molendijk - LLRF 2013

New RF Low Level system

MDDS DDC

DSP FPGA

VME 64xVXS

DDC DDC

DSP FPGA

DDC SDDS

DSP FPGA

Cavity driveC02, C04, C16Beam Phase Pickup

Transvers Pickups

4 x Σ & ∆ Gap returnsC02, C04, C16

B Field

Frequency Program& Radial loop

Beam Phase loop Cavity Drive & returns

Characteristics: Full PPM operation including cycle parameter archiving Unlimited inter DSP communication through Standard VXS [5] backplane All players DSP etc. have independent control interfaces DSP “Fast loop” cycle time 5 μs (DSP Clock 400 MHz + C programming). RF Clock and TAG distribution through VXS fabric Recent electronics (DSP FPGAs etc.) Standard FMC [7] mezzanine solution using powerful carrier FPGA resources (DSP specialized Virtex 5 XC5VSX95T) Firmware built-in remote flashing capability for both the FPGAs as well as the DSP code

Injection & extraction synchro loops

Ext. references

Hardware Architecture

7

VXS [5] DSP FMC-Carrier

+ +

= VXS-DSP-FMC-Carrier

J.C. Molendijk - LLRF 2013

VXS Standard VXS = VMEbus Switched Serial Standard

Standard VITA 41.0 [5] (2006 - now) Dual star backplane interconnect from payload slots to switch slot(s)

High-speed Multi-Gig RT-2 connectors P0 High bandwidth up-to 6.4 Gb/s supported Low latency point to point Reduced real-estate for interconnects


SwitchA

SwitchB

PayloadPort 1

PayloadPort 2

PayloadPort 3

PayloadPort 7

PayloadPort 6

PayloadPort 5

PayloadPort 4

PayloadPort 8

VXS Topology

VXS backplane


2 (A and B) x 4 full-duplex Giga-bit Serial links per payload slot Switch Slot A connects to all “A” links of all Payload slots. Similar for switch slot B 16 full-duplex interconnects between both Switch Slots (not shown)

A B

A

B

B&W Picture source: VITA AV41DOT0

VXS Boards


The High-speed Multi-Gig RT-2 connectors support: 8 Channel full-duplex High-speed differential signaling + I2C [6] Control link (Payload to Switch board)

No VME interface

B&W Pictures source: VITA AV41DOT0

VXS Switch Boards


3

X-bar Switch 72 x 72M21141

FPGAXC5VLX50T-1FF1136C

4 4 4 4

e-SATAADCLK948BCPZ

RF Clock1

ADCLK948BCPZ RF TAG1

e-SATA3

3

RF Clock Input

RF Clock Outputse-SATAe-SATA

1

1

P5

P3

P4

P24

20

20

4

PP11

PP1,3,5,7,9

PP2,4,6,8,10

PP12

VXS

SP3

SP1

SP2

SP4

1

1

SW_SE(8-1)

VXS

I2C Controls12

SW_RX/TX1

GA(4-0),GAP

SYSRST

PEN

P1 Source Select

RF TAG'

RF Clock'

SFP4

SFPSFP

SFP

PPWR1

VPC

P5V0

Gnd

Precharge Voltage

Power

4

Legend

4 Full-Duplex Gigabit channels5

5 RF Clock / TAG differential pairs4

4 Full-Duplex Gigabit channels A to B Switch

44 Full-Duplex fiber optic channels

SP3 Switch Board Port 3 (Switch Interconnect)

PP11 Ports to Payload Board 11

Rear Connectors

Frontpanel Connectors

D(7-0)

A(9-0),C

Sn,R

w,D

Sn,S

ETn

STA

TS

RF Clock

TAG

TAG Double TAG

TAG Encoding 50% duty at RF Clock / 2

To ispPWR

3

1

White Rabbit Compatible portGeneral purpose SFP

Agnostic X-bar With Multicast

VXS Switch Boards


VXS Crate overview

VXS Switch Boards


e-SATA

RF Clock & TAGDistributors

RF Clock & TAG e-SATA3e-SATA

e-SATA

RF Clock & TAG1Front panelRF Clock and TAGinput

Front panelRF Clock and TAGoutput

X-bar Switch 72 x 72M21141

RF Clock & TAG1

1

0

1

Source Select

1

RF Clock & TAG

RF Clock & TAG12

RF Clock & TAG12

PP1-PP12

VXS B portsRF Clock and TAGin and outputs

Example X-bar route

Legend

RF Clock & TAG1 RF Clock & TAGpoint to point routingRF Clock & TAG1 Unused RF Clock & TAG routing

5

FPGAXC5VLX50T-1FF1136C

1

RF Clock & TAGRF Clock & TAGMonitoring

1

VXS Switch B

Typical Clock & TAG routing Configuration 1. The Payload ports with the MDDS is driving RF Clock and TAG towards the X-bar switch 2. RF Clock & TAG is driven to all used Payload ports by X-bar switch (Multicast) 3. RF Clock & TAG distributors (eSATA outputs) driven by X-bar switch

FMC Standard


FPGA Mezzanine Card standard VITA 57.1 [7] Two flavors

Low-pin count connector 160 pins = 34 diff pairs* (68 single-ended) single bank IO voltage.

High-pin count connector 400 pins = Bank A: 58 diff pairs* AND Bank B: 22 diff pairs* with separate IO voltage per Bank

FMC Bank A IO (and FPGA VCCO) programmable voltage is supplied by the FMC Carrier (Vadj) as requested by mezzanine IPMI [8] ROM.

FMC Bank B IO (and FPGA VCCO) is powered from the FMC mezzanine (VIO_B_M2C). Also declared by the mezzanine IPMI ROM.

Notes: * Or: single-ended = 2 * number of diff pairs IPMI = Intelligent Platform Management System

FMC Modules


High Pin Count FMCs Developed [11] ADC 16 bit 125 MS/s (DDC) DAC 16 bit 250 MS/s (SDDS) DDS (can be used as Master Direct Digital Synthesis)

Notes: DDC = Digital Down Converter SDDS = Slave Digital Direct Synthesizer

ADC (DDC) FMC


Characteristics: General purpose 4 channel, 125MS/s acquisition module with 16bits architecture. Provides signal conditioning with an analogue front-end featuring DC coupling, low

noise, low distortion and gain switching (equivalent to 3 LSBs). DC to 40MHz (oversampled) bandwidth. Can be extended by a factor of 10. SNR > 77dB (12.5 ENOB), SFDR > 70dB

DAC (SDDS) FMC


Characteristics • 2 x 16bit Dual DACs (AD9747), Fs <= 250MS/s -> 4 identical CHs

• DC coupled output, 40MHz analog bandwidth, peak output voltage 3.6 Vp-p

• 400-pin FMC connector for parallel data interfacing

• Low noise and low distortion amplifiers

• Compact front-panel for heat dissipation and 3-color status indication LEDs

• Unique PCB identification by silicon ID chip and a system monitor chip

• IPMI EEPROM with HW specific information (FMC type, Version number, operating voltage etc.)

• Programmable 18 dB analog gain switch < 30 ns for dynamic range shift • Measured wideband SFDR > 60 dB

DDS FMC


Characteristics: • High quality compact clock and synchronism generator. • Integrated random time jitter: 674 fs RMS • Can generate a clock signal from 62.5 MHz to 125 MHz, and the associated revolution

synchronism signal at any FRev sub-harmonic from 1 to 16535. • It features two independent channels, synchronized to the same reference, with

synchronization pulse (tag) generators. • 32bit DDS core: 232 mHz frequency step resolution. • Compatible with LPC and HPC FMC standard.

VXS DSP FMC Carrier


A VME64x board with VXS VXS bank A banks fully equipped with 4 full-duplex channels, bank B with 2 Raw link-speed either 2 Gb/s.

Two High-pin count FMC sites Sites have independently programmable Vadj 4 FPGA IO banks (~20 diff pairs/bank) per FMC site.

3 FPGA IO banks for FMC Bank A, 1 FPGA IO bank for FMC Bank B. 4 full-duplex Serial Giga-bit links to FMC_FPGA per FMC site.

Two Virtex5 FPGAs: XCVSX95T-1FF1136 & XC5VLX110T-1FF1136 FMC_FPGA hosts the FMC DSP intensive code, interfaces with both FMCs, communicates to the

Main_FPGA via 8 Serial Giga-bit links and a parallel bus. Main_FPGA manages the communication with: VXS, FMC_FPGA, VME64x, DSP.

A Sharc DSP 400MHz: ADSP-21369 A16 / D32 interface with Main_FPGA. Alternate Serial ports connected to Main_FPGA.

Memory Banks Two 4 M x 18 @ 100 MSPS (CY7C1472V33) Two 1 M x 4 x 18 @ F-RF MSPS (CY7C1474V33)

RF Clock and Tag distribution.

FMC FPGA Main FPGA DSP

FMCs

VXS DSP FMC Carrier


8

Legend

8 Full-Duplex Gigabit channelsParallel Data

0 = GTP Dual Tile Y index. Ex: GTP_DUAL_X0Y0

P1

P2

P0

VME 64x

VXS

FMC0

6

SRAM 14Mx18

eSA

TA

Front

RF Clock& Tag

A(31-24)D(31-16)

FPGAXC5VLX110T-1FF1136

1A

Main

FPGAXC5VSX95T-1FF1136

1

A

FMC4

FMC2

4

DSPADSP-21369KBPZ-3A

Triggers I+O(15-0)

A(23-1)D(15-0)etc.

160

160 852

90

90

14

e-W

ISH

BO

NE

Tile 6, 7

Tile 4, 5

Tile(0:3)

Tile(4:7)Tile(0:3)

0

12

01 2

SRAM 11Mx4x18

SRAM 01Mx4x18

SRAM 04Mx18

200

CY7C1474V33-167BGC

CY7C1472V33-167BGC

RF+Tag

JTAG

JTA

G

PWRClocks

JTAG

F0 OKF2 OKSTAT

EDA-02281

RTM I2C

2

RF Clock & TAG

Power Supplies & Trigger in / out Via RTM

6 full duplex 2Gb/s channels + RF Clock & TAG

VXS DSP FMC Carrier Powering


DSPADSP-21369KBPZ-3A

2 * FPGAXC5VLX110T-1FFG1136CXC5VSX95T-1FFG1136C

VCCO_P3V3

VCCint_P1V010A

PTH04T240W

3ATPS74401KTWT

VCCAUX_P2V5

6A VCCOFMC0 Bank APTH04T230

6A VCCOFMC2 Bank APTH04T230

VADJ0 / VCCO_PxVy

VADJ2 / VCCO_PxVy

3APTH04T260 MGTAVCC_P1V0

3APTH04T260

MGTAVCCPLL / MGTAVTTTX /MGTAVTTRX / MGTAVTTRXC_P1V2

GTP

VDDint_P1V3

VDDext_P3V3

VME_P5V0

Max 10.8 A

2 * FPGA_CONFIGXCF32P

VCCO / VCCJ_P3V3

0.8ALP3961 1V8

VCCint_P1V8

FMCboth

VME_P12V0

3P3V_P3V3

3P3VAUX_P3V3

12P0V_P12V0IPS7081SPBF

FMCboth

FMC0_VIO_B_M2C

FMC2_VIO_B_M2C

VCCO_PxVy

VCCO_PxVy

Legend+12V+5V+3.3V+2.5V+1.8V+1.3V+1.2V+1.0V+1.2 / 1.5 / 1.8 / 2.5 / 3.3V

from FMCto Carrier

3APTH04T260W

ispPAC-POWR1014A

FMC FPGA Control

FMC FPGA Control

FMC FPGA Control

V-PrimaryMonitor

Max 12 A

Max 1.5 A

MAX1230BCTI+V-SecondaryI-PrimaryMonitor

SPI Interface5

I2C Interface2

5

5

autotrack

FDMC8296

FDMC8296

FDMC8296

2

H

H

H

H = Charge Pump Drive (+12V = "on")

OC

OC

OC

OC = Open Collector Drive

LL = Logic Level Drive (5V = "on")

16 channel

VME_P3V3

LMP8645MK

LMP8645MK

LMP8645MK

FMC FPFA Power Domains


Giga-bit Interface Pins in white

VME Interface

Main FPGA

Wishbone Master (Cntrl)

FPGA DSP Bridge

Wishbone Slave (Cntrl)

IP Core FMC 0


Gigabit Serial

SRAMintfce

SushiTrain Gigabit Serial

D32 / A16 Multiplexed INTERCON

VME D32 / A32

Core Addr Dec

D32 / A16 A32

DSP bus D32 / A32


CRegs

to be ANDed with Slave STB_I

FMC FPGA

e-Wishbone e-Wishbonee-WISHBONE

STB

_0

STB

_2

D32 / A16 Multiplexed INTERCON

FMC 0 FMC 2A BA B

34106 2 34106

Main

VXS P0

2

B A

VME 64x

I2C I2C

FMC Carrier

2

DSP

IP Core FMC 2


Gigabit Serial

SRAMintfce

SRAM1Mx4x18

SRAM1Mx4x18

100 100

SRAM4Mx18

SRAM4Mx18

45 45

to Main

Full-rate Parallel Storage

Reduced-rate Interleaved Storage

2 * Gigabit Serial

Clock & TAG

Functions

Timing

OASIS

Gigabit Serial Gigabit SerialFGC+Tim

DSP2FMC / FMC2DSP

11

2Legend

2 Full-Duplex 32 bit Gigabit channelsParallel Data

RF Clock & TAG1

Firmware Architecture Communications



Wishbone SoC1 Interconnection Architecture for Portable IP Cores [9]

• Simplifies design reuse & integration • Common standardized GP interface • Decouples IP cores • Divide & Rule: more control interfaces easier to manage • Easily extendable to seamlessly interconnect FPGAs • Enables individual software driver (per core) support

N Slaves Master

VME

Main FPGA

DSP

Switch Control

FMC FPGA

FMC 0 IP-core

FMC 2 IP-core

VXS-DSP-FMC CarrierMemory map

0

1

4

5

6

7

WishboneSlaves each64k x 32 bit

IndividualMemory maps /Register controlper WB slave

VME

1 SoC: System on Chip


Memory Map management The Cheburashka memory map configuration tool [10] has been created to achieve:

• Common definition of memory maps for FPGAs, CPU Software, DSPs • Automatically generate the memory map VHDL package as well as the register control

block VHDL (using GENA [10]). • Automatic creation of the Linux CPU device driver configuration • Automatic creation of a debug FESA (middleware) class • Support for construction nested (by reference) memory maps • Support for interface reversal (read-only to read-write and vice-versa) • Automatic creation of DSP header files • Integrated documentation: XML files from Cheburashka can be displayed on a web

browser

Project Status


• Hardware • Laboratory and successive tests on CERN PSB Ring 4 with a 2 board system,

built in V1 hardware (VXS-DSP-FMC Carrier and VXS-Switch) have successfully demonstrated the feasibility of the new digital Low-level RF system.

• The pre-series of V2 hardware has been validated • At present the 3 board system (V2 hw) has been transferred to a MedAustron

test stand for specific MedAustron software developments • Firmware

• The FPGA firmware developments are nearing completion • Remote updating of FPGA firmware and DSP now available • IPMI for FMC developments has started

• Software • Test DSP code has demonstrated successful. Code now well advanced for the

MedAustron / PSB system • The device drivers and test FESA classes generated with help from

Cheburashka work nicely in synchronism with the firmware • A control room application will be created this year.

Planning (non exhaustive)


• MedAustron • Cavity test-stand commissioning, Oct. Nov. 2013 • System Deployment in Austria Feb. 2014 onwards

• PS Booster 4 rings • System deployment and commissioning from April 2014 • Finemet tests from late 2014 and onwards

• LEIR • Upgrade to the new LLRF in 2015

• ELENA1 LLRF + Longitudinal diagnostics + Beam orbit system • Systems to be deployed and commissioned in 2016

• AD2 LLRF + Longitudinal diagnostics system • Upgrade to the new hardware in >= 2017

• …

1 ELENA: Extra Low ENergy Antiproton synchrotron [13] 2 AD: Antiproton Decelerator [14]

Conclusion


• The developed standardized hardware opens the way to many alternative applications beyond the scope of LLRF alone

• The novel modular approach using the wishbone bus has shown very efficient in a multi developer environment

• The tool used to guarantee coherent memory-maps between the software and hardware world has been essential for the success of this project

• The automatic register bank VHDL code generation tool is saving lots of time allowing to largely cut the firmware development time

• The built in remote FPGA and DSP reconfiguration tool simplifies machine development and operation

1 ELENA: Extra Low ENergy Antiproton synchrotron

Thank you for your attention.


References


[1] J. DeLong et al., “Topology for a DSP-Based Beam Control System in the AGS Booster”, PAC’03, Portland, May 2003, p. 3338. [2] M. E. Angoletta et al., “Feasibility Tests of a New All-Digital Beam Control Scheme for LEIR”, CERN/AB-Note-2004-004-RF. [3] M. E. Angoletta et al., “PS Booster Beam Tests of the New Digital Control System for LEIR”, CERN/ABNote-2005-017-RF. [4] M. E. Angoletta et al., “Beam tests of a new digital Beam Control System for the CERN LEIR Accelerator“, PAC’05, Knoxville TN. [5] ANSI / VITA 41.0, “VXS VMEbus Switched Serial Standard” (http://www.vita.com/vxs.html). [6] NXP, “The I2C-bus Specification” (http://www.nxp.com/documents/other/39340011.pdf) [7] ANSI / VITA 57.1, “FPGA Mezzanine Card (FMC) Standard” (http://www.vita.com/fmc.html). [8] Intel ”Intelligent Platform Management Interface” (http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-specifications.html ) [9] OpenCores, “WISHBONE System-on-Chip (SoC)Interconnection Architecture for Portable IP Cores” (www.opencores.org). [10] A. Rey, F. Dubouchet, M.Jaussi, T. Levens, J. Molendijk, A. Pashnin CERN, “A tool for consistent memory map configuration across hardware and software” ICALEPCS 2013, San Francisco CA. [11] P. Leinonen, J. Sanchez Quesada(CERN) , “FPGA Mezzanine Card standard IO-modules for the LLRF beam control system of CERN’s PS Booster and MedAustron synchrotron”, RFTech 2013, Annecy. [12] M. Benedikt, “MedAustron – the Austrian Ion Therapy and Research Centre” (http://cas.web.cern.ch/cas/Slovakia-2012/Lectures/Fabich.pdf) [13] T. Eriksson (editor) et al., “ELENA, an Updated Cost and Feasibility Study”, CERN-BE-2010-029. [14] T. Eriksson et al., “Upgrades and Consolidation of the CERN AD for Operation During the Next Decades”, IPAC2013, Shanghai, China, May 2013, WEPEA063, p. 2654.

http://www.vita.com/vxs.html

http://www.nxp.com/documents/other/39340011.pdf

http://www.vita.com/fmc.html

http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-specifications.html



http://www.opencores.org/

http://cas.web.cern.ch/cas/Slovakia-2012/Lectures/Fabich.pdf



Two board test setup in CERN PSB Feb. 2013


DDC Overview


The down converter is an homodyne receiver that converts the selected beam revolution harmonic into a baseband I/Q signal.

It features a multi-rate

fast signal processing hardware embedded into the FPGA IP core.

DDC Overview


The narrowband dynamic range is controlled by an automatic gain control, maximizing the SNR when the input signal power does not fill the ADC range. It adds 18dB to the effective analog dynamic range.

Further improvement is obtained with process gain:

DAC FMC Overview


DDC Overview


The narrowband dynamic range is controlled by an automatic gain control, maximizing the SNR when the input signal power does not fill the ADC range. It adds 18dB to the effective analog dynamic range.

SDDS Main Characteristics


• Delay corrector, Azimuth, phase offset compensation and MDDS harmonic number change compensation features

• Selectable amplitude and phase modulations, and noise generation for the controlled beam emittance blow-up

• Comprehensive signal observation through SRAM memory

• Highly flexible signal generator with many features

Contains SPI, I2C, Memory, power supply and A/D gain processing controllers

Has a direct VME, DSP and memory access but also FMC to FMC IP core link, main FPGA to FMC FPGA link for status/faults etc., PG indication signals and many more…

Multiple debugging options

Amplifier offset compensation manually and automatically according to the preference

• User selectable manual/automatic amplifier DC offset compensation

SDDS Main Characteristics


All measurements are completely independent and unrelated

DAC mezzanine output

FFT of the 16-bit NCO signal ~ 5MHz, peak at 89dB (Matlab plot)

DAC mezzanine V1 during testing

-61dBm

71dB

+10dBm

SFDR

SFDR ~ 60dB between 3MHz – 40MHz

MDDS Overview


Simplified block diagram:

MDDS Operating Principles


The RF Clock follows the frequency program (Frev) at an harmonic such that the output frequency falls always between the maximum sampling frequency and half.

This scheme allows the homodyne system to track the selected Frev harmonic, while maintaining the sampling rate high enough to guarantee a minimum SNR.

MDDS Test Results


Divider change + Tag synch pulse 125MHz clock phase noise

Random jitter at the clock output vs output frequency B-Field frequency ramp

MDDS System Clock Distribution Context


System clock distribution context:


VXS DSP FMC Clock Distribution

PSB RFLL Upgrade

Documents

Transcript of PSB RFLL Upgrade