Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit...

Connect. Communicate. Collaborate

4 Gigabit Onsala - Jodrell Lightpath for e-VLBI

The iNetTest Unit

Development of Real Time eVLBI at Jodrell Bank Observatory

7th International eVLBI Workshop, Shanghai, 16-17th June 2008

Richard Hughes-Jones DANTE Jonathan Hargreaves JBO


The 4 Gig Path for EXPReSOnsala – Jodrell

Multi:• Domain• Vendor• Technology


First Test Path

4 Gig TDM/SDHLighpathover GÉANT2+

10 Gigabit Ethernet Lambdaover NORDUnet

Connect. Communicate. CollaborateConnect. Communicate. Collaborate

UDP Throughput Stoc-Lon• Alcatel Metro Core Connect MCC

Flow control OFF• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre

– rx-usecs=25 Coalescence ON

• MTU 9000 bytes• Max throughput 4.05 Gbit/s

• Packet loss as expected

• Sending host, 3 CPUs idle• For <8 µs packets,

1 CPU is >90% in kernel mode

• Receiving host 3 CPUs idle• For <20 µs packets,

1 CPU is ~30% in kernel mode

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

0 5 10 15 20 25 30 35 40Spacing between frames us

Recv W

ire r

ate

Mb

it/s

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8972 bytes

8000 bytes

Stoc2-1_10Jan08

0

20

40

60

80

100


% P

ac

ke

t lo

ss

1000 bytes

1472 bytes 2000 bytes




8000 bytes

Stoc2-1_10Jan08

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40

Spacing between frames us

%c

pu

1 k

ern

el

sn

d

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8972 bytes

8000 bytes

Stoc2-1_10Jan08

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40

Spacing between frames us

% c

pu

1 k

ern

el

rec

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8972 bytes

8000 bytes


UDP Jitter Stoc-Lon• Alcatel Metro Core Connect MCC

Flow control OFF• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre

– rx-usecs=0 Coalescence OFF

• 8972 byte packets• 16.5 μs spacing

• Peak in packet spacing at 16 μsas expected

• Other peaks at 19 and 51 µs

8972B w=16.5 sto-man_rx-usecs0_5Apr08

0

100000

200000

300000

400000

500000

600000

700000

0 20 40 60 80 100 120 140Latency us

N(t

)8972B w=16.5 sto-man_rx-usecs0_5Apr08

1

10

100

1000

10000

100000

1000000

0 20 40 60 80 100 120 140Latency us

N(t

)


exp1-Stoc2_10Jan08.xls

0

20

40

60

80

100


% P

ac

ke

t lo

ss

1000 bytes





8000 bytes


UDP Throughput Lon-Stoc• Alcatel Metro Core Connect MCC

Flow control ON• Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre


• MTU 9000 bytes• Max throughput 4.05 Gbit/s

• No Packet loss – flow control

• Sending host, 3 CPUs idle• For <18 µs packets,

1 CPU is >60% in kernel mode

• Receiving host 3 CPUs idle• For <20 µs packets,

1 CPU is ~30% in kernel mode

exp1-Stoc2_10Jan08.xls

0

20

40

60

80

100


%c

pu

1 k

ern

el

sn

d

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8972 bytes

8000 bytesexp1-Stoc2_10Jan08.xls

0

20

40

60

80

100


% c

pu

1 k

ern

el

rec

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8972 bytes

8000 bytes

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000


Recv W

ire r

ate

Mb

it/s

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8972 bytes

8000 bytes


TCP reno Throughput Lon-Stoc• Alcatel Metro Core Connect MCC

Flow control ON• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre


• MTU 9000 bytes• TCP buffer 32 MByte 2*BDP

• iperf Ave throughput 3.85 Gbit/s• iperf Max throughput 4.02 Gbit/s

• 3 re-transmitsDue to “other reductions”


TCP reno Throughput Stoc-Lon• MCC NO Flow control• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre



• iperf Ave throughput 192 Mbit/s• iperf Max throughput 465 Mbit/s

• TCP Cwnd to 0• Many re-transmits•


Alcatel MCC Buffer size

• Classic Bottleneck• 10 Gbit/s input 4 Gbit/s output• Use udpmon to send a stream of spaced UDP packets• Measure packet number of first lost frame as function of w packet spacing

)R*w(PNQ out1lostlen )/Q(R*wP/QN/1 lenoutlen1lost

Slope gives buffer size ~57 kBytes


Use UDP to emulate TCP slowstart

• udpmon sends bursts of spaced packets:– 32 packets– Jumbo 8000 bytes– back2back– 4 ms between bursts

• PathLon-Ams_FF-Prague-Paris-Lon

• Rtt 55.5 ms

• See 13 packets then loose 1 in 3

• Confirm the TCP problem!

0

5

10

15

20

25

30

0 10 20 30 40 50 60

packet number

1-w

ay t

ime


iNetTest: iBoB FPGA with 10 GE


iNetTest: Control & Operation

10/100 Ethernet

10Gb Port 0

iNetTest iBOB 1

Send and receive packets of selected length and spacing

Count packets transmitted and received, calculate histograms of arrival times

Control PC

IP control of multiple iBOBs

10Gb Port 1

10Gb Port 0

10Gb Port 1

Ten Gb Network

iNetTest iBOB n

Expansion to more than two iBOBs

iNetTest iBOB 2

Send and receive packets of selected length and spacing

Count packets transmitted and received, calculate histograms of arrival times


iNetTest: Simulink design

Jonathan HargreavesJBO

Network test parameter CSRs

10GE MAC core

Inter-packet timehistogram

Transmit & receive event time log

Connect. Communicate. CollaborateiNetTest: Details• iNetTest FPGA runs with a 200 MHz (5 ns) clocks • Two iNetTest units can be controlled over ethernet IP from a PC• iNetTest Ethernet IP address and MAC preset in code

– last digit selected by jumper• Ten GE IP address, Port and Gateway can be user configured• Automatic or manual ARPing. ARP tables for each port can be examined• iNetTest responds to and generates PING• UDP packets can be sent by firmware between two iNetTests

or between iNetTest and PCs• User selects the number of packets to send, their length,

and the time between them• iBOBs count received and transmitted packets, and store up to 2048 time

stamped events per port• iBOBs generate histograms of the arrival time distribution for received

packets


UDP Throughput vs. Packet Spacing

• PC• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R CX4


• MTU 9000 bytes• UDP Packets• Max throughput 9.4 Gbit/s

• iBoB• Packet 8234

Data: 8192+ Header: 42• 100 MHz clock• Max rate 6.6 Gbit/s• See 6.44Gbit/s• 200 MHz clock - Linespeed


PC to iNetTest Packet Jitter

• Kernel 2.6.20-web100_pktd-plus

• Myricom 10G-PCIE-8A-R Fibre

• Packet separation 100 µs

• iBoB bins 2320 ns

• Width ± 34 µs

• Similar to PC-PC but no extra peaks

• Structure caused by PC

8900 bytes w=100 exp1-2_rxcoal0_16May07

1

10

100

1000

10000

100000

0 50 100 150 200 250 300

Latency us

N(t

)

esleaPC-ibob_ipg_24Oct07

1

10

100

1000

10000

100000

1000000

10000000

0 50 100 150 200 250

Time between frames us

N(t

)

The PC – PC plots look like:


iNetTest – iBOB to iBOB Results

• 10 GE 3m CX4 cable link• 8192 byte packets• 1 million transmitted at line rate between two iNetTest units.• 5 ns bins

• B2B 10 ns FWHM• • With Switch:

– Fujutsu XG2000 20 ns– HP 6400cl 35 ns long tail 1

10

100

1000

10000

100000

1000000

6570

6575

6580

6585

6590

6595

6600

6605

6610

6615

6620

6625

6630

6635

6640

6645

6650

6655

6660

6665

6670

6675

6680

iBOB-iBOB

via Fujitsu XG2000

via HP6400cl flow ctrl ON

via HP6400cl flow ctrl OFF

Number of

Packets

Time between packet arrivals (ns)


Back to the 4 Gig Path


Stockholm – Manc. with TSS


4 Gigabit Ethernet Path over TSSNORDUnet

4 Gig on EthernetOptical TransmissionJANET & NNW


UDP Throughput Stoc-Man with TSS• Alcatel TSS – 10GE – MCC

• Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre


• MTU 9000 bytes

• Max throughput 3.78 Gbit/s(Previously was 4.05 Gbit/s)

• Packet loss 10 % at 4.096 Gbit/s for 8192 Bytes

• Man-Stoc:No packet loss @ 4.096 Gbit/s

sto-gig6_rx75_TSS_21May08

0

1000

2000

3000

4000

5000

6000

7000


Rec

v W

ire

rate

Mb

it/s

1000 bytes

1472 bytes

2000 bytes

3000 bytes

4000 bytes

5000 bytes

6000 bytes

7000 bytes

8000 bytes

8972 bytes

sto-gig6_rx75_TSS_21May08

0

20

40

60

80

100


% P

acke

t lo

ss

1000 bytes





8972 bytes


Packet loss Stoc-Man with TSS• Alcatel TSS – 10GE – MCC

• Send 1M packets from PC in Stockholm

• 8192 bytes

• All cross the TSS cloud• All enter Alcatel MCC

as 10 GE frames

• Packet loss in MCC Card • ~10% at 4096 Mbit/s

• Classic “bottleneck” performance

• But WHY???

0

2

4

6

8

10

12

0 1000 2000 3000 4000 5000

Offered Rate Mbit/s

% P

acke

t L

oss


UDP Jitter Man-Stoc with TSS• Alcatel MCC – 10GE – TSS – PC • Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre


• 8192 byte packets• Alcatel MCC is TDM 28 VC-4

Can only send packets at 16 µsNot at 10 Gbit/s

• Peak packet spacing ~ 6 µs ie packets arriving at 10 Gbit/s

• Suggests packet bunching in the network

8192B w=16 gig6-sto_rx0_19Jun08

050000

100000150000200000

250000300000

350000400000

0 20 40 60 80 100 120 140Latency us

N(t

)

8192B w=16 gig6-sto_rx0_19Jun08

1

10

100

1000

10000

100000

1000000

0 20 40 60 80 100 120 140Latency us

N(t

)8972B w=16.5 sto-man_rx-usecs0_5Apr08

0

100000

200000

300000

400000

500000

600000

700000

0 20 40 60 80 100 120 140Latency us

N(t

)

Sto-Man no TSS16 µs


UDP 1-way delay Man-Stoc with TSS• Alcatel MCC – 10GE – TSS – PC • Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre


• 8192 byte packets•

• Structure similar to interrupt coalescence but this is OFF

• As function of recv timesee gaps of ~ 130 uspackets at ~ 10Gbit/s

• Again suggests packet bunching in the network

W16 gig6-sto_rx0_19Jun08

18600

18650

18700

18750

18800

18850

18900

18950

19000

0 50 100 150 200Packet No.

1-w

ay d

elay

us

W16 gig6-sto_rx0_19Jun08

1860018650

1870018750

1880018850

1890018950

19000

75580000 75585000 75590000 75595000 75600000receive time us.

1-w

ay d

elay

us


Last week: Onsala – JodrelliNetTest - iNetTest


4 Gigabit Ethernet Path over TSSNORDUnet

4 Gig on EthernetOptical TransmissionJANET & NNW


iNetTest: Onsala JBOUDP Packet Jitter

• 1M packets• 8192 byte packets• 16 µs spacing 4.096 Gbit/s• • 10 % packet loss

• Peaks 14 & 15.7 µs• Tail to 130 µs

• Cyclic variations spacing3.25 µs

• 45 packets at 6.5 µs (line rate)

0

50000

100000

150000

200000

250000

0 50 100 150Packet to Packet arrival time (us)

Num

ber

of P

acke

ts

1

10

100

1000

10000

100000

1000000

0 50 100 150Packet to Packet arrival time (us)

Num

ber

of P

acke

ts


Development of Real Time eVLBI at JBO

eMERLIN Import – Onsala to Jodrell eMERLIN Export – Jodrell to JIVE


Overview of eVLBI at JBO

eMERLIN CORRELATOR

JBO JIVE

4Gbps light path

OnsalaJBOCX4 4GbpsOr fibre if > 15m

CX4 4Gbps iBOB 5SwitchiBOB 4 Switch ADCStation Board VSI

VSI

4Gbps light

path

CX4 1Gbps

CX4 1Gbps

CX4 1Gbps

CX4 1Gbps

iBOB 0 SwitchStation Board VSI

VSI

VSI to ZDOK

VSI to ZDOK

Switch

iBOB 1Station Board VSI

VSI

VSI to ZDOK

VSI to ZDOK


VSI

VSI to ZDOK

VSI to ZDOK


VSI

VSI to ZDOK

VSI to ZDOK

VLBI Mk V b receiv

ers


Onsala to JBO: Transmit eMERLIN Import

ZDOK0

iADC

2 channels

1024MSPS per channel

8 bit sampling

10GB PORT

. 4GB/s out

ASSEMBLE PACKETS

Divide data into 8192 byte packets plus 16 byte MkVC compatible header

Alternatively generate test packets as per iNetTest

Count packets transmitted

DATA Left Pol

ADCCLK

ONEPPS

Convert data to 2 bit

Measure signal power on both channels

Generate real time clock based on ADC clock

Pass data & sync from ADC clock to iBOB 200MHz clock domain

Power PC

iBOB control and monitoring via registers on the OBP bus

10/100 ethernet

Measured RMS Signal Power

Read and Set Real Time Clock

Set up test mode parameters

Control Data Flow ON/OFF

Set Network Source and Destination

DATA Right Pol

iBOB


Onsala to JBO: Receive

ZDOK0

10GB PORT

. 4GB/s in

RE-ORDER PACKETS

Store data in SRAM location according to its time stamp and sequence number

Regenerate Sync and Data Valid signals

FIFO

16k deep

Smooth out bunching due to network delay variations

Power PC


10/100 ethernet

iBOB

STREAM OUT

Clock data out of SRAM on 128MHz correlator clock

Stream the left and right polarisations to the Station Board VSI chips

ZDOK1

VSI Chip 0

Route left polarisation data to input stage of Station Board

Send correlator clock to iBOB

VSI Chip 1

Route right polarisation data to input stage of Station Board

Station Board

SRAM0

512 packets L pol. data

SRAM1

512 packets R pol. data

DATA 32 bit

DATA 32 bit

128MHz clock

Connect. Communicate. CollaborateOnsala to JBO - Status

• iADC and transmit iBOB tested with ADC clocked at 1GS/s in diagnostic mode – ADC is set to output a fixed bit pattern

• iBOB receiver and VSI chip coded in Simulink and VHDL

Next Steps

• Test iADC and transmit iBOB with sinusoidal test signal• Generated deliberately out-of-order packets and bursts of

greater than 4Gbps to test receiver buffering


JBO to JIVE: Transmit – OvervieweMERLIN Export

ZDOK0

10GB PORT

. 1 Gb/s out

REMOVE DELAY MODEL

(COURSE)

Use 72 bit wide bt 512k deep SRAM to remove eMERLIN delay model with 16 microsecond resolution

FORMAT

Generate 10000 byte MkVb frames

Power PC


10/100 ethernet

iBOB

ZDOK1

VSI Chip 0

Process left polarisation data

Detail on next slide

VSI Chip 1

Process right polarisation data

Detail on next slide

Station Board

DATA 32 bit

DATA 32 bit

CLOCK

VALID

PDATA

CLOCK

VALID

PACKETISE

Divide each MkVb frame into two 5008 byte packets (incl header)

Add Mk5C header to each packet


JBO to JIVE: Transmit -VSI Chip Detail

VSI Chip 0 - Process left polarisation data

Monitor & Control Bus (MCB)

Station Board

DATA 32 bit

CLOCK

VALID

PDATA

Select a 128MHz

band from the

filter bank

Subtract Merlin Delay (Fine)

Variable coefficient FIR filter changes delay in steps of 1/16th of original sample period

Reclock for delays up to 16ns

Mixer 0

FIRLow pass filter

Decimate by 8/16128 taps

Convert 4/8 bit data to 2 bit VLBI

Mixer 1

Mixer 8/16

Stream data out to iBOB over VSI

cable

MCB InterfaceLoad band selection, delay model and filter

parameters

0

1

15

Mixer Bank

Select 8 or 16 VLBI bands from

128MHz input

Remove eMERLIN offset

(a multiple of 10kHz)

Connect. Communicate. CollaborateJBO to JIVE: Status

• VSI chip DSP functions currently under development and simulation

• Transmit iBOB re-uses packetising code from iNetTest

• Next Steps• VSI chip MCB interface – based on Verilog code provided

by Dave Fort at Penticton• Transmit iBOB needs coarse delay and MkVb formatting


Summary

• UDP on a 28 VC-4, 4.2 Gigabit lightpath provides the performance required for EXPReS

• TCP performance poor due to small Ethernet buffer size on the equipment connecting 10GE to the 4 Gig Lightpath

• Ethernet flow-control with short tails helps TCP

• The network behaves correctly.• Care needed on choice of PC for 10 Gigabit.• FPGA iNetTest solutions work very well.• Onsala – Jodrell 4 Gigabit path now in place.• Packet loss and packet spacing need more understanding. • VLBI FPGA application progressing well.


Any Questions ?

Connect. Communicate. CollaborateFor More Information

• www.geant2.net• www.dante.net• For latest news and factsheets http://www.geant2.net/media

• For research activities http://www.geant2.net/research• http://expres-eu.org/ [note: only one “s”]• http://www.jive.nl/ • Contact information:

Richard Hughes-Jones, DANTE, [email protected] Hargreaves, JBO, [email protected]

• EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #02664


GÉANT2 Topology

November 2006


TCP BIC Throughput Lon-Stoc• Alcatel Metro Core Connect MCC

Flow control ON• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre



• iperf Ave throughput 3.98 Gbit/s• iperf Max throughput 4.02 Gbit/s

• Several re-transmits•


TCP BIC Throughput Stoc-Lon• MCC NO Flow control• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre



• iperf Ave throughput 71.6 Mbit/s• iperf Max throughput 192 Mbit/s

• TCP Cwnd rapidly opens• Many more re-transmits•


TCP BIC Throughput Lon-StocBut Faster and Longer

• MCC Flow control• 48 VC-4• RTT 60 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre


• MTU 9000 bytes• TCP buffer 59 MByte 1 * BDP• txquelen 1300• …

• iperf Ave throughput 6.5 Gbit/s• iperf Max throughput ~6.9 Gbit/s

• Some glitches & re-transmits•

Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit...

Documents

Transcript of Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit...