5 - 8 January 2009National Radio Science Meetings1 The ALMA Data Transmission System – Digital...

5 - 8 January 2009 National Radio Science Meetings 1

The ALMA Data Transmission System – Digital Portion

Chris Langley

ALMA Back End Integrated Product Team


The Challenge

Transmit 4 – 12 GHz

Astronomical Data from the Front End (FE) Band

Cartridges to the Correlator using Commercial Off The Shelf equipment wherever possible.


The Challenge

Transmit 4 – 12 GHz

Astronomical Data from the Front End (FE) Band

Cartridges to the Correlator using Commercial Off The Shelf equipment wherever possible.

The Proposal

Convert FE data Digitally and Optically prior to transmission from each of 66 antennas.


The Flaw

COTS, or any other, D/A converters capable of

4 – 12 GHz inputs were not available during R&D.


The Flaw

COTS, or any other, D/A converters capable of

4 – 12 GHz inputs were not available during R&D.

The SolutionSeparate, or Down Convert, the 4 – 12 GHz two

polarity band into eight 2 – 4 GHz basebands prior to data conversion and transmission.


Astronomical Data Down Conversion & Transmission

P0 IFDCP0

LSB

Virtual Parallel Bus

10 Gb/s X 12≤ 15 Km

DTX 0

DTX 1

DTX 2

DTX 3 DRX 3

DRX 2

DRX 1

DRX 0BB 0

BB 1

BB 2

BB 3

BB 4

BB 5

BB 6

BB 7

BB 0

BB 2

BB 4

BB 6

BB 1

BB 3

BB 5

BB 7

BB 0

BB 1

BB 2

BB 3

BB 4

BB 5

BB 6

BB 7

P1 IFDC

4-12 GHz

USB

USB

LSB

P1

2-4 GHz

256 bits @ 125 MHz

Front End

Correlator


Data Transmission System

P0 IFDCP0LSB

Virtual Parallel Bus

10 Gb/s X 12≤ 15 Km

DTX 0

DTX 1

DTX 2

DTX 3 DRX 3

DRX 2

DRX 1

DRX 0BB 0

BB 1

BB 2

BB 3

BB 4

BB 5

BB 6

BB 7

BB 0

BB 2

BB 4

BB 6

BB 1

BB 3

BB 5

BB 7

BB 0

BB 1

BB 2

BB 3

BB 4

BB 5

BB 6

BB 7

P1 IFDC

4-12 GHz

USB

USB

LSBP1

2-4 GHz

256 bits @ 125 MHz

Front End

Correlator


Design Considerations (1/2)

• Operate at OC-192 (9.95328Gb/s) optical fiber signaling speed




• Use of time division digital multiplexing to transform the input signaling rate to the channel signaling rate





• Insertion of fill bits to convert input rate to signaling rate





• Insertion of fill bits to convert input rate to signaling rate• Use of time division digital de-multiplexing to transform

the channel signaling rate to the output signaling rate






the channel signaling rate to the output signaling rate• Elimination of un-needed fill bits upon reception






the channel signaling rate to the output signaling rate• Elimination of un-needed fill bits upon reception• Use of three OC-192 channels per 2 GHz baseband to

achieve required capacity



• Low-voltage differential signaling (LVDS)– Fast rise/fall times– Noise resistant




• Multiple FPGA design per channel– More economical than single FPGA– Ball Grid Array package Lots of IO– 625+ MHz input signal capability




• Multiple FPGA design per channel– More economical than single FPGA– Ball Grid Array package– 625+ MHz input signal capability

• Commercial Optical “Half” Transponders– Change from original design– Became economical– Built in mux /demux, clock recovery




• Multiple FPGA design per channel– More economical than single FPGA– Ball Grid Array package– 625+ MHz input signal capability

• Commercial Optical “Half” Transponders– Economical– Built in mux /demux, clock recovery

• Air cooled (flow through) module, RFI shielded (-50 dBm)


Data Frame Organization

6 bits 5 bits 124 bits 16 bits4 bits4 bits

BIT

16

BIT

20

BIT

0

L

SB

(fir

st b

it)

BIT

6B

IT 7

LS

B

BIT

12

L

SB

of

pa

ylo

ad

BIT

14

4

L

SB

(fir

st b

it)

BIT

15

9

M

SB

(la

st b

it)

M

SB

of

pa

ylo

ad

LS

B’s

of

syn

c p

att

ern

Me

tafr

am

e in

de

x (1

bit)

Se

qu

en

ce c

ou

nt

4 L

SB

’s o

f p

ayl

oa

d

MS

B’s

of

syn

c p

att

ern

12

4 M

SB

’s o

f p

ayl

oa

d

che

cksu

m


Data Transmission SystemCloser View

Formatter &Optical

Transmitters

FiberOpticMUX

3-bit Digitizer

P0

3-bit Digitizer

P1

4 GHz

FiberOpticDe-

Mutiplexer

DRX 0Optical

Receivers &De-

Formatter

3-bits X 16@ 250 MHz

X4D (+/-)

D (+/-)

C

C

B

B

2 – 4 GHzP0

2 – 4 GHzP1

1-bit X 10 Gb/s X 12

DTX 1

DTX 2

DTX 3

EDFA

120 Gb/s

DRX 1

DRX 2

DRX 3

DTX 0

1-bit X 10 Gb/s X 12

3-bits X 32@ 125 MHz

X4


Data Transmitter Module(Digitizer and Formatter, 4 per Antenna)


Data Transmitter Module4 / Antenna

250 MHz, 15.6 psec stepped delay, 0 dBm

48ms, LVDS

4 GHz, 15.6 psec stepped delay, 0 dBm

125 MHz, 0 dBm

P0

P1

From IFP

From DGCKLRU

3 bitA/D

3 bitA/D

1:16Demux

1:16Demux

1:16Demux

1:16Demux

1:16Demux

1:16Demux

AMB 0x5X, X = 0, 1, 2, 3From ABM

FPGAB

FPGAD

FPGAC

TXP

TXP

TXP

AMBSI-2

SPI

SPI DG

To FOM

B Bits

D Bits

C Bits

DG FR

MCPS

Backplane

5.0 VDC

15.0 VDC

Power Harness

PCB

-5.2 VDC

3.3 VDC

PWR FILTERSFrom PSAC

48 VDC

3.3

VD

C

5.0

VD

C

-5.2

VD

C

3.3

VD

C

5.0

VD

C

15

.0 V

DC

48 VDC

From FOMOptical Keep Alive

2.13-3.95 GHz0 +/- 1 dBm

- 10 dB

- 10 dB

P0

P1

ITU Wavelength

21, 27, 33, or 39

ITU Wavelength

25, 31, 37, or 43

ITU Wavelength

23, 29, 35 or 41


Digitizer AssemblyUniversity of Bordeaux


Formatter with 3 Optical Transmitting Transponders


Data Receiver Module(De-Formatter with 3 Optical Receiving Transponders)


Data Receiver Module4 / Antenna

AMBSI-2

FPGAAltera EP1S20F780C5 D Bits

RX Transponder with 1:16 DeMux

From FOAD

156.25 MHz

Data (D Bits)

16 bits @ 625 Mb/s

Recovered Data Clk

625 MHz

Steering Clk

FPGAAltera EP1S20F780C5 C Bits


From FOAD

156.25 MHz

Data (C Bits)

16 bits @ 625 Mb/s

Recovered Data Clk

625 MHz

Steering Clk

FPGAAltera EP1S20F780C5 B Bits


From FOAD

156.25 MHz

I2C Bus

Control Bits

Data (B Bits)

16 bits @ 625 Mb/s

Recovered Data Clk

625 MHz

Steering Clk

JTAG ChainSPI

-5.2 VDC

1.8 VDC

3.3 VDC

5.0 VDC

-5.2 VDC

1.8 VDC

3.3 VDC

5.0 VDC

-5.2 VDC

1.8 VDC

3.3 VDC

5.0 VDC

DC Power Regulation1

.5 V

DC

-5.2

VD

C

1.8

VD

C

3.3

VD

C

5.0

VD

C

5.0 VDC 3.3 VDC

48 VDC

SPI

SPI

JTAG Chain

JTAG ChainFPGA Comm.

FPGA Comm.

1.5 VDC

3.3 VDC

1.5 VDC

3.3 VDC

1.5 VDC

3.3 VDC

From Station Bin Motherboard

125 MHz

48 ms

10 Gb/s

10 Gb/s

10 Gb/s

3.3 VDC

P0

P1

P0

P1

P0

P1

32 bits each @ 125 MHz



Configuration EEPROM Altera EPC16QC100

AMB

To / From Correlator Station Bin Motherboard


DTS Modules for 1 Antenna

Digitizer Clock

IRAM, NRAO

Data Transmitters

U of Bordeaux,NRAO

Fiber Optic Multiplexer

Jodrell Bank Observatory

Fiber Optic Amplifier /

Demultiplexer

Jodrell Bank Observatory

Data Receivers

NRAO


DTS Link Tests - ALMA Antenna to LabChile, 8/2008


Things We’d Do Differently …

• Single FPGA per channel!– FPGA logic timing is difficult– Economics will likely catch up

• Closer interaction between hardware and firmware designers– Each should be the other’s backup

• Invite external expert’s opinions sooner during the design process

• Test Stand– Design and build once assembly form factors are determined

• Communication between remote team members was good, but could have been better– Specify an early DTS design review for the international partners


Acknowledgements

• Robert Freund, Principle Engineer, Arizona Radio Observatory

• Paula Metzner, DTS Product Engineer, Atacama Large Millimeter Array, National Radio Astronomy Observatory

… and the entire DTS teams from North America, the University of Bordeaux, IRAM (Grenoble, FR), and Jodrell Bank Observatory (~Manchester, UK).

R. W. Freund, ALMA Memo 420: Digital Transmission System Signaling Protocol, 2002

R. W. Freund and C. Langley, BE Critical Design Review, 2004.

References


Auxiliary Slides


Data Transmission System Overview The Partners

UB Effort

UK Effort

US Effort

To FOAD#2 - #50

To FIR/Correlator

Fibers

10 GB/s Fibers

Cable

15 Km

~185 Similar CablesAntennas & Sites

Control Building

Cables Fibers

12 1

50 49

~186 Cables

LO M/C Fibers

1

50

Fibers

12

1

Fiber

DG

(4)

FRTXT(4)

FOMux

FOPatchPanel

SpliceRack

FOADAnt #1

DFRRXT(4)

LO, M/C Fibers

FromIFP,2 - 4 GHz,Two

Polarities

2 base bands x 96 bits

@ 125 MHz

Antennas 2- 50

Antenna 1

Positions for 49 more Antennas

120 GB/s Fibers


System Requirements

• Repeatable latency with no loss of samples


System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)


System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s


System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source


System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source• Nyquist sampled data


System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source• Nyquist sampled data• 3-bit data word


System Requirements

• Repeatable latency with no loss of samples• Bit error rate < 10-6 (End of Life)• Multi-channel synchronization loss < 10-4 s• 16 GHz analog bandwidth source• Nyquist sampled data• 3-bit data word• Data transmission synchronized with ALMA

timing


System OverviewExplicit requirements

• 4 GSa/s per 2 GHz bandwidth IF channel



• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample



• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels



• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels• 96 Gb/s per antenna (120 Gb/s encoded data)



• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels• 96 Gb/s per antenna (120 Gb/s encoded data)• 250 MHz input word rate (96-bit wide parallel word)



• 4 GSa/s per 2 GHz bandwidth IF channel• 3 bits per sample• 2 Polarizations x 4 IF channels• 96 Gb/s per antenna (120 Gb/s encoded data)• 250 MHz input word rate (96-bit wide parallel word)• 125 MHz output word rate (192-bit wide parallel

word)




word)• Grouping of a polarization pair: 24 Gb/s per pair




word)• Grouping of a polarization pair: 24 Gb/s per pair• Walsh function 180° switching




word)• Grouping of a polarization pair: 24 Gb/s per pair• Walsh function 180° switching• 15 Km (maximum) distance


System OverviewImplied requirements

• Configurable if not deterministic timing (repeatable latency)




• Fast frame synchronization




• Fast frame synchronization• Continuous transmission of data




• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life




• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life• Operation independent of payload content




• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life• Operation independent of payload content• Testing strategies




• Fast frame synchronization• Continuous transmission of data• Low error rates throughout operational life• Operation independent of payload content• Testing strategies• Economical implementation


Auxilary Slides

DTS Single Bit Data Path(VHDL Top Level)


Frame Implementation

• 160 bit frame• 128-bit payload• 10-bit synchronization word• 5-bit meta-frame sequence word• 1-bit meta-frame index• 16-bit odd parity check word


Design Decisions

• Use of a data block (frame) to facilitate multiplexing


Design Decisions

• Use of a data block (frame) to facilitate multiplexing• Use of a synchronization (framing) word to facilitate

frame detection


Design Decisions


frame detection• Use of scrambling techniques to minimize bit sequence

effects, low frequency content, and to maintain signal balance


Design Decisions




• Use of a meta-frame to synchronize the reception of multiple channels


Design Decisions





• Use of a meta-frame index to synchronize reception to ALMA timing under varying propagation delays


Design Decisions





• Use of a meta-frame index to synchronize reception to ALMA timing under varying propagation delays

• Use of a checksum word to facilitate continuous monitoring of received data integrity


Auxilary Slides

DTX Formatter FPGAs and Transponders

625 MHz

625 MHz

625 MHz

625 MHz

625 MHz

625 MHz

FIBER10 GBs

FIBER10 GBs

FIBER10 GBs

TTX FRAME DATA D

TTX FRAME DATA C

TTX FRAME DATA B

TTX 1

TTX 2

TTX 3

u100

u200

u300

ADC

D

C

B

TTX SPI

DATA

CLK

SPI

DATA

DATA

CLK

CLK

SPI

SPI

SPI

TE

GE

NT

E G

EN

AM

BS

I SP

IA

MB

SI S

PI

FR

AM

E

SY

NC

FR

AM

E

SY

NC

BB0

BB1

FRAMECLK

BB0

BB1

BB0

BB1

FRAMECLK

FRAMECLK

SPI

DIGITIZER SPI

TESYS

SYSCLK

AMBSI

DIGITIZER

CLOCK DIVIDER

BB0A

BB1A

4GHz

250MHzphase shifted

16

16

16

16

16

16

16


Auxilary Slides

DRX De-Formatter FPGAs and Transponders

BB0

BB1

BB0

BB1

BB0

BB1625 MHz

625 MHz

625 MHz

FIBER10 GBs

FIBER10 GBs

FIBER10 GBs

TRX MUX’D DATA D

TRX MUX’D DATA C

TRX MUX’D DATA B

TRX 1

TRX 2

TRX 3 TOP_C

MID_B

BOT_A

EEPROM

D

C

B

TRX I2C

DATA

CLK

I2C

DATA

DATA

CLK

CLK

I2C

I2CSPI

TE

GE

NT

E G

EN

AM

BS

I SP

IA

MB

SI S

PI

FR

AM

E S

YN

CF

RA

ME

SY

NC

SYSCLK

AMBSI

16

16

16

FR

AM

E C

OU

NT

FR

AM

E C

OU

NT

SP

EC

DE

LAY

REF CLK

REF CLK

REF CLK

32

32

32

32

32

32

CORRELATORDATA

196 BITS @ 125 MHz

TE

4


Auxiliary Slides

Frame Synchronizations

►Frame synchronization (10-bit synchronization word)

• Multi-channel synchronization (5-bit meta-frame sequence word)

• Determination of propagation delay (1-bit meta-frame index)


Auxilary Slides


• Frame synchronization (10-bit synchronization word)• Multi-channel synchronization (5-bit meta-frame

sequence word)• Determination of propagation delay (1-bit meta-frame

index)


Auxiliary Slides

Frame synchronization (10-bit synchronization word)

• Unique or “unique enough” pattern to minimize acceptance of erroneous patterns in random data

• Long enough pattern to eliminate the acceptance of erroneous pattern in static data

• Partitioned pattern to eliminate the acceptance of a correct pattern in an incorrectly configured system

• Three acceptance stages required to qualify a 10-bit quantity as the synchronization pattern


Auxiliary Slides

Stages for Frame Synchronization

• Search:

selection of an initial location within the serial bit-stream followed by the shifting of the location until a candidate synchronization word is located

• Check:

continued observations in subsequent frames until unsuccessful criterion (failure)

• Monitor:

once confirmed, continuous monitoring of all frames to ensure proper operation


Auxiliary Slides


• Frame synchronization (10-bit synchronization word)

►Multi-channel synchronization (5-bit meta-frame sequence word)

• Determination of propagation delay (1-bit meta-frame index)


Auxiliary Slides

Multi-channel synchronization (5-bit meta-frame sequence word)

• Sequence word large enough to accommodate worst case relative variation in propagation delay across the three channels

• Integer number of meta-frames contained within one 48.000ms timing period

• Transmitter simultaneously writes the identical incrementing sequence number in frames of all three channels

• Receiver compares and re-times the frames from the three channels thus synchronizing the meta-frames


Auxiliary Slides


• Frame synchronization (10-bit synchronization word)

• Multi-channel synchronization (5-bit meta-frame sequence word)

►Determination of propagation delay (1-bit meta-frame index)


Auxiliary Slides

Determination of propagation delay (1-bit meta-frame index)

• Transmitter uniquely identifies the first meta-frame following a 48.000ms timing event

• Monitor and Control system obtains the count of frames received following the local 48.000ms timing event and the detection of the meta-frame index bit

• Monitor and Control system command the receiver to adjust its internal frame delay to a specific relative value


Auxilary Slides

Scrambling

• Modification of source data to accommodate specific characteristics of the communication channel

• Provides adequate timing for the clock and data recovery electronics

• Provides a signal balance for the AC coupled circuits which minimizes threshold errors

• Pattern is easily produced by a maximally length shift register generator

• Sync word is exempt from scrambling• For 149 -> 0, Result <= output pattern (shifted +1) XOR

input pattern


Auxilary Slides

Data Integrity

• Parity computation is easier than CRC• 16-bit parity word over 144 bits• Each parity bit monitors 9 other bits• Permits continuous monitoring of transmission quality


Auxiliary Slides

Self Test Methods

• 10 GHz clock recovery• Frame detection• Multiple channel synchronization• Scrambled pattern exercises• Random Number Generation• Checksum (parity) checks• FFT of pseudo Front End data (gain flatness,

CW beacon)


Formatter Block Diagram

Protocol Encoder

HalfTransponder

Protocol Encoder

HalfTransponder

Protocol Encoder

HalfTransponder

Sixteen10:1 Mux’s

Sixteen10:1 Mux’s

Sixteen10:1 Mux’s

1:4 Demux

1:4 Demux

1:4 Demux

250 Mb/s 625 Mb/s62.5 Mb/s 62.5 Mb/s 10 Gb/s(1bit optical)

20.833 Hz

125 MHz

(128 bits) (160 bits) (16bits)

MCEngine

X5MultiplierDC-DC

Converters

Monitor PointA/D’s & Mux’s

(32 bits)

FPGA 1

FPGA 2

FPGA 3

P0

P1

P0

P1

P0

P1

3.3 VDC

From Digitizer Assembly

From DGCK

To FOM

To / From Digitizer Assembly

From MC / PS

To / From MC / PSBoard

-5.2 VDC

5 VDC

15 VDC 625 MHz

Laser Keep Alive

From FOM


Auxilary Slides

Digitizer Clock Module


Auxilary Slides

Digitizer Clock AssemblyIRAM, Grenoble


Auxilary Slides

Fiber Optic MultiplexerJodrell Bank Observatory


Auxilary Slides

Fiber Optic Amplifier / Demultiplexer


Auxilary Slides

Transmitter Module – Internal View

Formatter

Digitizers

Monitor Control &Power Supply

Backplane


Auxilary Slides

DTS Test Stand

• “Golden” Modules– 2 Data Transmitters– 1 Digitizer Clock– 2 Data Receivers

• Support Electronics– PC with LabView

Interface– System timing– Data Receiver

backplane






• Test Stand– Design and build once assembly form factors are determined

5 - 8 January 2009National Radio Science Meetings1 The ALMA Data Transmission System – Digital...

Documents

Transcript of 5 - 8 January 2009National Radio Science Meetings1 The ALMA Data Transmission System – Digital...