LHC – LEIR approach Present LEIR Hardware Possible upgrades Planning

A.Blas AB/RF/FBA.Blas AB/RF/FB Team meeting 21/1/2008Team meeting 21/1/2008 11

Upgrade possibilities Upgrade possibilities for thefor the Hardware Hardwareof the of the LEIR type Beam ControlLEIR type Beam Control

Information fromInformation from::M.E. Angoletta, P. Baudrenghien, E, Bracke, A. Butterworth,M.E. Angoletta, P. Baudrenghien, E, Bracke, A. Butterworth,

J. Ferreira, J. Molendijk, R. Olsen, F. Pedersen, T. Rohlev, J. SanchezJ. Ferreira, J. Molendijk, R. Olsen, F. Pedersen, T. Rohlev, J. Sanchez

1. LHC – LEIR approach2. Present LEIR Hardware3. Possible upgrades4. Planning


Beam Control Beam Control Typical architectureTypical architecture

Both the LHC and LEIR system start with the same initial structure

Typical constraint for the signal processing delay:Loop computation time + Hdw delay < 1/(8.Fmod) [phase lag @ max Fmod < π/4]

< 21 us for PSB & LEIR phase loop (3.Fs=6kHz)


Beam Control Beam Control Signal processingSignal processing

In LEIR the FPGA-DSP-FPGA triptych is always used

In LHC it depends.

Note that using the DSP in a loop implies major timing problems when a precise loop delay is required (TFB, 1TFB) (see example in Elettra)


Beam Control Beam Control Board layoutBoard layout

Interconnect architecture shared by the LHC and LEIR hardware

The Central FPGA (fpga array in Leir) routs the data from and to the different locations

•The special function board can be in the form of an on-board specific circuit, a daughter cards or a dedicated VME board


Beam Control Beam Control ClockingClocking

LHC:A central VCO, controlled by a central processor, outputs the rf (400MHz +/- ) feeding the cavity. This rf signal is used to create the 380, 80, 40, 20, Frev which are dispatched within the hardware.

LEIR: A central DDS, controlled by a central processor, outputs the tagged rf clock distributed to all rf sources or demod inputs.(+) Absolute phase control over distant inputs/outputs without (except practical) limitations on their number.(+) Synchronous change of parameters on all boards.(+) The Clock harmonic can be changed on the fly (factor 1, 2, 4 and 8) for use on a wide frequency span (factor 16)(+) synchronous signal acquisition


Beam Control Beam Control Hardware standardsHardware standards

LHC: VME 64 crate, h: 9U 4 front slots, 160 mm depth with standard P1/P2 bus 15 front slots, 220 mm depth with dedicated P2 bus Spec. power supplies setup 6 kCHF + 10 kCHF CPU

LEIR: VME 64X crate, h: 9U 23 front slots, 160 mm depth 23 back slots, 160 mm depth All standard P0, P1, P2 connectors Standard Power supply 8 kCHF + 10 kCHF CPU


Samtec QSE, 80 (4 small ribons of 20 coax cables on a single connector

160 Gb/s (at 2 GHz)

3.2 Gb/s (at 40 MHz)

Used in LHC hardware

Gigabit serial link with 8b/10b coding16bit// 50 MHz -> 1Gb/s serial

800Mb/sFull duplex at the same rate

Used in LHC hardwareVery common standard

DSP bus + mother board to daughter card bus at 40 MHz and 32 bit word: 1.28 Gb/s (to be divided by the number of wait states +2 = 4)

320 Mb/s Used in LEIRDSP <-> DC

Link port clocked at 40 MHz (can be used at 80 MHz)

320 Mb/s Used in LEIRDSP <-> DSP

Beam Control Beam Control Communication linksCommunication links


FPGA comparisonFPGA comparison

XilinxXilinx

Virtex 2Virtex 2

XC2V2000-XC2V2000-

6FG676C6FG676C

(Tuner loop)(Tuner loop)

Altera StratixAltera Stratix

EP1S20FEP1S20F

484C5484C5

(Leir (Leir daughter daughter cards)cards)

XilinxXilinx

Virtex 4Virtex 4

XC4VLX40XC4VLX40

Beam PhBeam Ph

+ rf Mod. + rf Mod.

AlteraAltera

Stratix IIStratix II

EP2S90FEP2S90F

1020 C41020 C4

LHC 1TFBLHC 1TFB

Logic cellsLogic cells 24,19224,192 18,46018,460 41,47241,472 90,96090,960

RAM [kb]RAM [kb] 1,0001,000 1,6001,600 17281728 44144414

18 x 18 multipliers18 x 18 multipliers 5656 8080 6464 192192

I/OI/O 456456 361361 640640 758758

Price [USD]Price [USD] 278278 350350 267267 28902890


DSPs comparisonDSPs comparison

ModelModel ClockClock

[MHz][MHz]

MMACSMMACS LinkLink

portsports

SerialSerial

LinksLinks

(not used)(not used)

Unit priceUnit price

for 1000for 1000

(USD)(USD)

ADSP-21160MADSP-21160M

(Leir)(Leir)

8080 160160 66

80 MHz max80 MHz max

22


159159

ADSP-21160NADSP-21160N

(upgrade Leir)(upgrade Leir)

100100 200200 66


22


170170

ADSP-TS101SADSP-TS101S

(LHC Tuner loop)(LHC Tuner loop)

300300 21002100 44


Incompatible with Sharc DSP!?Incompatible with Sharc DSP!?

Dual edge = 250 MHz eqDual edge = 250 MHz eq

00 186186

ADSP-TS201S

(available novelty)(available novelty)

600 48004800 44


00 242242


DSP chipDSP chip ADSP 21160M (Leir at present)

80 MHz, 4Mbit on chip SRAM, 2.5V core


DSP chipDSP chip Possible LEIR upgrade ADSP 21160N

Same as M version except 20% faster (100 MHz) and 1.8V core


DSP chipDSP chip Tiger Sharc ADSP-TS101 (LHC)


Leir BC <-> LHC BCLeir BC <-> LHC BC

LHCLHC LEIRLEIRModified VME 64 crateModified VME 64 crate

160 mm (CO)+ 220 mm (rf)160 mm (CO)+ 220 mm (rf)

Private Backplane bus with:Private Backplane bus with:

Power, ECL Clocks, Timing, Function Power, ECL Clocks, Timing, Function Gen, Interlocks, Alarms, LVDS data bus, Gen, Interlocks, Alarms, LVDS data bus, JTAG, Auto slot addressing, Module serial JTAG, Auto slot addressing, Module serial number bus number bus

Standard VME 64X crateStandard VME 64X crate

160mm with rear access to bus160mm with rear access to bus

Cadence blocks:Cadence blocks:

Tiger sharc + environmentTiger sharc + environment

Virtex 4 + environmentVirtex 4 + environment

1 Gb serial link + environment1 Gb serial link + environment

Tagged clock + distributionTagged clock + distribution

(connectors are fragile; why??)(connectors are fragile; why??)

RTM for timing, data, clock, link portsRTM for timing, data, clock, link ports

High speed parallel connection between High speed parallel connection between VME neighborsVME neighbors

Samtec QSE 80 // coax linesSamtec QSE 80 // coax lines

Mother-daughter card structure Mother-daughter card structure interconnected via the DSP businterconnected via the DSP bus


Leir BC <-> LHC BCLeir BC <-> LHC BC

LEIRPros:•Standard VME 64X crate with front and rear access•Very modular approach; with 3 DC (DDC, SDDS, MDDS), 1 MB and 1 RTM, you have the base for all type of applications (except yet for 1TFB and TFB)•Synchronous control over all the hardware

LHCPros:•Power supplies adapted to low noise requirements•More up-to-date circuits and connectors


Leir Beam Control Leir Beam Control Clock coreClock core

Output rf frequency <12 MHz (for a swept frequency - without up-conversion)<25 MHz at fixed frequency (without up-conversion)

Limitations due to the basic structure (max ADC DAC sampling + tag creation circuit)


Leir Beam Control Leir Beam Control AcquisitionAcquisition


Leir Beam Control Leir Beam Control OutputsOutputs


Beam Control Beam Control General architecture (LEIR)General architecture (LEIR)


Beam Control Constraints Beam Control Constraints Data flowData flow

Rule of thumb:

The loop will be sufficiently stable if its delay leads to a phase lag < /4 at the unity loop gain frequency

Loop computation time + Hdw delay < [1/(8.Fmod)]< 21 us for PSB & LEIR (3.Fs=6kHz)< 26 us for PS (3.Fs=4.8 kHz)

In LEIR, the in-loop DSP is sampling the data every TS-DSP = 12.5 us (80 kHz).The loop delay within the DSP = import data from DDC (<50ns) + compute error (<7us) + send error to MDDS or SDDS (via another DSP or not < 150 ns) + equivalent 1st order S/H delay (6.25 us) => ≈14us delay within the DSPWith a 80 kHz DSP sampling clock, an averaging (CIC) of 1000 80-MHz-samples in the DDC would be adequate. We actually use 256, which means <6.4 us extra delay. This means that we are approaching the reasonable limits required for the LEIR and PSB phase loop.

The DSP process time in LEIR is the most time consuming and multiplying by a factor 2 this process speed would almost double the possible bandwidth.


Leir Beam Control Leir Beam Control Data FlowData Flow

Here, the central DSP B has all its links ports used (limiting factor) In case of more DSP boards, we would need to add links to the leading DSP or implement a pipe lined communication protocol through DSP boards or make all DSPs share a single bus with a communication protocol to be defined. In terms of bandwidth, A gigabit link could replace the link ports and the DSP-to-daughter card link


Leir Beam ControlLeir Beam ControlDaughter cardsDaughter cards

The present DDC (ADC) and SDDS (DAC) daughter cards having 4 channels are limited in terms of FPGA logic cells.The signal monitoring circuit could not be implemented.The 4 channels of the DDC need to share 2 LOsThe DDC FIR could not be implemented

The present chip: Altera Stratix EP1S20F 484C5 should be replaced with a version with 6 Altera Stratix EP1S20F 484C5 should be replaced with a version with 6 times more logic elements to have a 50% loading at most. 18k LE -> 110 k LE and twice as times more logic elements to have a 50% loading at most. 18k LE -> 110 k LE and twice as much RAM (1.6 Mb -> 3.2 Mb) and same I/O: 361.much RAM (1.6 Mb -> 3.2 Mb) and same I/O: 361.

The Stratix 3 The Stratix 3 EP3SL150 can do the job: 142 kLE, 5.5 Mb, 480 or 736 I/O, 1400 euros (Spoerle)


DSP boardDSP boardBackground

04/2002 Original design by Joe Delong (BNL)

03/2005 Original version imported in the CERN database (EDA 00990) and called version 113 units constructed and tested. Followed-up by J. Ferreira-Bento.

08/2006 Upgrade to version 3 by Tony Rohlev, after a faulty version 2which had a bad set of connections on a FPGAMain improvements: latched registers interfacing the VME bus + adequate power-up sequence. (see AB-Note-2007-031-RF for a detailed description of the modifications)

09/2007 Version 4. Cleaned-up version : removed 4 daughter card connectors (only 2 DC slots now), 1 FPGAs, the Event link circuit, the link-port connections to the DC connectors (reflections), Added DSP address bus lines 15-19 to the DC to avoid paging of the

SRAM address + allow use of DMA (DMA lines added) + use of delayed SELN bits for the register address decoding, all 16 RTM signal connected (6 before).


DSP boardDSP board Version 1 Block Diagram

VME Data [31,0]Bi-Dir

Transceiver

VME interface

FPGA

Bi-DirTransceiver

DSP Data [63,32]

Mem Data [63,32]

DIROE

DSP Addr[31,0]

DSP Data[32]

DSP Control

VME Data [7,0]

VME Addr [31,1]

Tri-stateBuffer

L,L,VME Addr [31,2] Mem Addr [31,0]Bit 31,30 always at 0

MEAS SRAM256k x 32b

X 4MemAddr [17,0]

Mem selectEPLD

MemAddr [31,18]

DSP controlMS[3,0]

SEL[3,0]Selects 1 of the 4 banks

DSP SRAM256k x 32b

X 4

Mem selectEPLD

DSP controlMS[3,0]


H,Mem Addr [31,19]

DSP Addr[18,1]

DSP Data[63,32]

DSPFlash

1M x 8bDSP Addr[19,0]

DSP Data[39,32]

OE

OE

DSP2RCVFPGA

DSP Addr[31,0]

DSP Data[63,32]

DSP Control

DSP

DSP Addr[31,0]

DSP Data[63,0]

DSP Control

Site SEL[7,0]

Site connector

1+2or 3+4DSP Control

DSP Addr[14,0]

DSP Data[63,32]

Site SEL

2 link ports F.P.

4 link ports VME P2

4 link portsSite connectors

Timing int.FPGA

DSP Addr[31,0]

DSP Data[63,32]

Trig[5,0] from VME P2

DSP Data[39,32]

Daughter card SRAM1M x 16b

Daughter card FPGA

DSP Addr[14,0]LSB always H

DSP Data[63,32]

RAMAddr[19,0]

RAMData[15,0]

Tri-stateBuffer

L,DSP Addr [31,1] Mem Addr [31,0]Bit 31 always at 0

OE

Bi-DirTransceiver

DSP Data [31,0] Mem Data [31,0]Bit 31 always at 0

OEDSPWRH

DIR

DIR

Problem: When the DSP SRAMis controlled by the VME busThe DSP is blocked


DSP boardDSP board Version 3 Block Diagram

VME interface

FPGA

DSP Data [63,32]

Mem Data [63,32]

DSP Addr[31,0]

DSP Data[32]

DSP Control

VME Data [7,0]

VME Addr [31,1]

Mem Addr [31,0]Bit 31,30 always at 0

MEAS SRAM256k x 32b

X 4MemAddr [17,0]

Mem selectEPLD

MemAddr [31,18]

DSP controlMS[3,0]


DSP SRAM256k x 32b

X 4

Mem selectEPLD

DSP controlMS[3,0]


H,Mem Addr [31,19]

DSP Addr[18,1]

DSP Data[63,32]

DSPFlash

1M x 8bDSP Addr[19,0]

DSP Data[39,32]

DSP2RCVFPGA

DSP Addr[31,0]

DSP Data[63,32]

DSP Control

DSP

DSP Addr[31,0]

DSP Data[63,0]

DSP Control

Site SEL[7,0]

Site connector

1+2or 3+4DSP Control

DSP Addr[14,0]

DSP Data[63,32]

Site SEL

2 link ports F.P.

4 link ports VME P2

4 link portsSite connectors

Timing int.FPGA

DSP Addr[31,0]

DSP Data[63,32]

Trig[5,0] from VME P2

DSP Data[39,32]

Daughter card SRAM

1M x 16b

Daughter card FPGA

DSP Addr[14,0]LSB always H

DSP Data[63,32]

RAMAddr[19,0]

RAMData[15,0]

Tri-stateBuffer

L,DSP Addr [31,1] Mem Addr [31,0]Bit 31 always at 0

OE

Bi-DirTransceiver

DSP Data [31,0] Mem Data [31,0]Bit 31 always at 0

OE

VME Data [31,0]Bi-Dir

latchedTransceiver

Bi-Dir latched

Transceiver

OE, DIR, LE

Tri-statelatched Buffer

L,L,VME Addr [31,2]

OE, DIR, LE

OE,LE

DSPWRH

DIR

Here the DSP SRAM data can be storedon the VME bus to allow the “long” ~1usVME Read process while letting the DSP run


DSP boardDSP board Some limitations

1. DSP blocked when the VME (which has priority) accesses the DSP SRAM (improved in version 3), Problem solved in Leir with pre-defined time slots for each access. Not convenient for long cycling machine as AD where operational data should be accessed during the cycle. Bus lines arbitration by 5 (4 in V4) distinct FPGAs => not flexible and uneasy to maintain

2. DSP limited computing power. Impairs the rf loops pipe-line delay and signal quality (aliases),requires assembly coding (instead of a more universal C coding)

3. DSP configuration device not accessible via VME for remote programming, nor daughter card JTAG programming.

4. Only 6 timing (or digital) inputs from J2/P2 connector (RTM allows 16) (solved in V4)

5. 4 MB DSP memory + 4 MB for measurement memory foreseen to be too little (-> 32 MB total)

6. VME base address selected by FPGA coding (-> put switches also)


LHC tuner loopLHC tuner loopDSP boardDSP board

The Tiger Sharc (tested) circuit can be re-used. For the FPGA there might be better choices in other circuit using Virtex 4.


Proposed upgrade for the LEIR type DSP mother boardProposed upgrade for the LEIR type DSP mother boardMaria Elena’s Maria Elena’s notenote

1.1. Single FPGA interconnecting all buses, timing, resets and clocks.Single FPGA interconnecting all buses, timing, resets and clocks.

2.2. Fast VME/DSP arbitration for memory access.Fast VME/DSP arbitration for memory access.

3.3. Additional FPGA as link port multiplexer.Additional FPGA as link port multiplexer.

4.4. Upgrade / doubling of the DSP chipUpgrade / doubling of the DSP chip

5.5. Increase memory size (4x -> 32 MB)Increase memory size (4x -> 32 MB)

6.6. Additional High speed inter-processor links Additional High speed inter-processor links

7.7. Increase the number of lines between RTM and J2/P2Increase the number of lines between RTM and J2/P2

8.8. Misc….Misc….

Other possible Other possible improvementsimprovements

1.1. Add a couple of gigabit serial links from central FPGA to front panel. This would allow Add a couple of gigabit serial links from central FPGA to front panel. This would allow using future specific boards as we do presently with daughter cardsusing future specific boards as we do presently with daughter cards

2.2. A DAC could be connected to the Central FPGA for monitoring/test purposesA DAC could be connected to the Central FPGA for monitoring/test purposes

3.3. Provide a connection port to a stand alone VME daughter card (Samteq QSE) same Provide a connection port to a stand alone VME daughter card (Samteq QSE) same as for LHC boardsas for LHC boards

4.4. VME offset address selected by jumpers instead of hardcodingVME offset address selected by jumpers instead of hardcoding

5.5. Suppress link ports lines to daughter card sockets (electric reflections) (done V4)Suppress link ports lines to daughter card sockets (electric reflections) (done V4)

6.6. Increase the address bus width connecting the daughter cards to avoid paging (DMA) Increase the address bus width connecting the daughter cards to avoid paging (DMA) and avoid the delayed SELN address decoding bit (done V4)and avoid the delayed SELN address decoding bit (done V4)

7.7. Use better connectors for the tagged clock distributionUse better connectors for the tagged clock distribution

8.8. Implement a communication protocol to chain DSP boards to ease Beam control Implement a communication protocol to chain DSP boards to ease Beam control upgrades and avoid piling-up of cables (critical in the CPS)upgrades and avoid piling-up of cables (critical in the CPS)

9.9. JTAG remote programming as in the CO with a dedicated FPGA-EPROM pair used JTAG remote programming as in the CO with a dedicated FPGA-EPROM pair used as a VME-to-JTAG interface. All the FPGAs should be in a single JTAG loop as a VME-to-JTAG interface. All the FPGAs should be in a single JTAG loop accessible on the front panel for local programmingaccessible on the front panel for local programming..


Leir Type DSP BC upgrade.Leir Type DSP BC upgrade. SummarySummary

Some CadenceTM blocks can be used from LHC designs with improved specs (tiger shark) + Gigabit link + general interconnect philosophy

The Virtex 4 or Stratix 2 circuits could also be imported but we should move to Virtex 5 or Stratix 3 (lower power dissipation 65nm, serial link interface not bugged + higher performances). To be used on DSP board and daughter cards.

The standard VME 64X remains a good choice. Noise issues have still to be checked for AD (linear Power Supplies).

The tagged clock and its distribution are a real plus for synchronous operation but the actual connectors are too fragile and could be replaced the RJ45 model used in LHC rf.

Add all the hardware details mentioned on the previous slide.


Leir Type DSP BC upgrade.Leir Type DSP BC upgrade.Possible scenarioPossible scenario

Install a test beam control in the PSB for the 2008 run, using our present stock of Hardware. A few units are still to be tested. Can be ready in mid 2008.

Analysis of the software requirements if using a new DSP (MEA)

Definition of an adequate inter-DSP link (h/w + protocol) (1man.month)

Update the mother board with all the new features keeping the compatibility with the present daughter cards. 8 man-month for the hardware, 6 man-month for the software + 40 kCHF. Ideally the software should be finished before we start the hardware.

Update of the daughter boards (DDC + SDDS): 3 man-month x 2 for the hardware + 1 man-month x 2 for the software + 40 CHF -> 1 working prototype for each function.

Total: 2 man-year + 80 kCHF for a “final” DSP beam control prototype hopefully fulfilling the requirements of all the Injectors on the Meyrin site.


DSP BC upgradeDSP BC upgrade Generic viewGeneric view


Master DDSon DSPboard B

Cavity

RF

2*RF

RFtoI/Q

H2Cavity

PID

H4Cavity

PIDPhase h4/h2

/toI/QCavity

Azimuth

2nd harmonic

h2Cavity Voltage

Voltageratioh4/h2

I/QtoRF

I/Qto

2*RF

/toI/Q

2

PIDControl

Two otherchannels

for cavity 2

Two otherchannels

for cavity 2

B TRAIN

Binj, Bej

PU 1Left

Steering

Frequency offset

Freq Prog (500 kHz)

20

2 BB

BFFREV

RadialPID

PIDControl

RFtoI/Q

PU 1Right

RFtoI/Q

LR

LR

PU 2Left

RFtoI/Q

PU 2Right

RFtoI/Q

LR

LR

Monitoring

4 CH receiveron DSP board A

4 CH receiveron DSP board B

DSP board B

BeamPick-up

I/Qto

Angle

PhasePID

RFtoI/Q

StablePhase

PIDControl

RFtoI/Q

REFRF

CAVSum

I/Qto

Angle

SyncPID

SynchroPhase

PIDControl

MonitoringDDSClock

1 GHz

h

Limiter

Tagged Clockat h*FREV

Taggingat FREV

4 CH receiveron DSP board C

DSP board C

4 ch modulatoron DSP board C

2*RFtoI/Q

DSP board A

1st harmonic

TaggedClock

TaggedClock

TaggedClock

TaggedClock

Freq.discri

CriticalProcess

time

CriticalProcess

time

Leir Leir Beam Beam ControlControl


MiscMisc

Arctan computation:FPGA Cordic John: 17 clock periods for a 16 bit resolution DSP Pawel: 33 clock period for a 18 bit resolution

LHC – LEIR approach Present LEIR Hardware Possible upgrades Planning

Documents

Transcript of LHC – LEIR approach Present LEIR Hardware Possible upgrades Planning