Recent eVLBI developments at JIVE

EVN-NREN meeting, Schiphol, 28-1-2005, A. Szomoru, JIVE

Recent eVLBI developments at JIVE

Arpad Szomoru

Joint Institute for VLBI in Europe


Outline• Introduction• Protocols, tuning issues• Tests and results

• ftp-based• Dual buffered• Single buffered• Real-time

• Jumbo frames• Network monitoring• Centralised control


Introduction•Move to disk-based

operations nearly complete•More reliable, data quality•Cheaper to maintain•Ease of use•Direct access compared•Unattended operations

•eVLBI: the future?•No consumables•High bandwidth•Fast turn-around

Disk based recording

eVLBI using fiber


• for the EVN and JIVE • Feasibility of eVLBI:- Costs, timescales, logistics.• Standards:- Protocols, parameter tuning, procedures at

telescope and correlator.• New Capabilities:- Higher data rates, improved reliability,

quicker response.

• for GÉANT and the NRENs• To see significant network usage with multiple Gbit streams

converging on JIVE.• Minimum is three telescopes (not including Westerbork)• Must be seen to enable new science and not just solve an

existing data-transport problem• Real-time operation is seen as the ultimate aim, buffered

operation accepted as a development stage.

POC targets:


Fibre pair from Amsterdam

Optical Splitters

GE lines to correlator

LX Optics converters

Cisco ONS 15252

Correlator Interface

Current status:•JIVE: 6 lambdas via Netherlight, each capable of 1 Gbps

•1 Gbps connections to Westerbork, Torun, Onsala, (Metsahovi?), 2.5 to Jodrell

•155 Mbps to Arecibo

•Connection of Medicina at 1 Gbps planned later this year


Protocols• Use of existing protocols, available hardware

• TCP: maximal reliability, very sensitive to congestion

• UDP: connectionless, fast(er) but not reliable (enough?)

• Different protocols will be tested in the near future (tsunami?)


Tuning issues• Dramatic improvement of TCP performance

possible by adjusting default settings:

• congestion window: product of roundtrip time and bandwidth

• size of queues between kernel and NIC• SACK (Selective Acknowledgment) implementation• MTU size (jumbo frames)• Interrupt moderation: cpu may be bottleneck

http://sravot.home.cern.ch/sravot


Tuning issues• Dramatic improvement of TCP performance

possible by adjusting default settings:

• congestion window: product of roundtrip time and bandwidth

• size of queues between kernel and NIC• SACK (Selective Acknowledgment) implementation• MTU size (jumbo frames)• Interrupt moderation: cpu may be bottleneck


Mem-mem transfer via patch

200

400

600

800

1000M

bps

Dell- SMJumboSM-Dell


Disk2net2disk via patch, jumbo

300

400

500

600

0,00E+00 2,00E+06 4,00E+06

workbuf (bytes)

Mbp

s 26832020971524194304


Disk2net-net2disk Dwingeloo-Bologna

0

100

200

300

400

Dw-BoBo-DwDw-Bo*Bo-Dw*


Network stress test (iperf)

0

200

400

600

800

Dw-BoBo-Dw

Dw-TrTr-Dw

Dw-OnOn-Dw

Mbp

s


Iperf (TCP) 30s intervals

0

100

200

300

400

Mbp

s

Tr-DwDw-Tr


• Intel(R) SE7501BR2• Intel(R) 7501 chip set• Dual Intel(R) Xeon (single,

2.8G, 512K L2 cash installed)• Dual channel DDR266

registered SDRAM (512MB)• Dual peer

PCIX 64/100, 2+2 slots32/33 PCI, 2 slots

• On-board TX1000 and T10/100 NICs

• On-board Ultra320 SCSI• On-board PATA

Mark 5 Motherboard Upgrade


Memory-memory Disk2net-net2disk In2net-net2disk

In2net-net2out

UDP TCP UDP TCP TCP TCPBench via patch 930 250 256Idem, jumbo frames 960 544 512Bench via Amsterdam

500 360 256

Idem, jumbo frames 341 456Westerbork-JIVE 867 680 256 256Idem, jumbo frames 249 378Bologna-JIVE 670 128 307Jodrell-JIVE 50 70 64 256Arecibo-JIVE 88 32Torun-JIVE 800 260 256Onsala-JIVE 177 256 256JIVE-Haystack 612 71

Test results


Huygens descent tracking• Data recorded at Mopra and Parkes, flown to

Sydney, transported via dedicated 1Gbps lightpath to Jive

• Peak transfer rate ~400Mbps• Use of modified tsunami protocol

• “slow start” algorithm• Transfer freezes occasionally• Quite buggy (but not everybody thinks so..)


eVLBI Transfer modes• ftp-based: bandwidth not critical, has greatly

improved response to technical problems at telescopes

• Dual buffered: recorded on Mk5 diskpack, transferred through disk2net and net2disk, played back at correlator

• Single buffered: streamed directly from formatters to diskpacks at JIVE

• Real-time: directly from formatters to correlator without disk buffering


eVLBI Transfer modes• ftp-based: bandwidth not critical, has greatly

improved response to technical problems at telescopes

• Dual buffered: recorded on Mk5 diskpack, transferred through disk2net and net2disk, played back at correlator

• Single buffered: streamed directly from formatters to diskpacks at JIVE

• Real-time: directly from formatters to correlator without disk buffering

First real-time eVLBI Image


MERLIN, March 2002 eVLBI, September 2004

IRC+10420


Jumbo frames• Tested with:

• Onsala @4470• Torun @8192• Westerbork @9000

• Results inconclusive, but…• Transfer between Mk5’s (bench)

• UDP: <500 Mbps• TCP: >500 Mbps, ONLY with jumbo frames

• Other hardware platform needed?


Network monitoring

• BWCTL installed at most stations• More measuring points along the way needed• Script-based tools for inspection and

visualisation of state of network


Centralised control• New operating model for EVN; correlator as

integral part of network


Centralised control• New operating model for EVN; correlator as

integral part of network• Monitoring tools at stations• Reliability of correlator, stability of Mark5 units• Choice of platform?


Conclusions• Mark5 not very well suited for eVLBI

• Hardware limitations (interrupt conflicts, outdated motherboards, proprietary hardware)

• Software limitation (stuck to kernel 2.4 because of Jungo drivers, proprietary software)

• Next platform?• support for 4Gbps, PCI Xpres?• as much off-the-shelf as possible• as simple as possible (no disks!)

• Dedicated lightpaths…

Recent eVLBI developments at JIVE

Documents

Transcript of Recent eVLBI developments at JIVE