JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard...

JIVE VLBI Network Meeting 15 Jan 2003R. Hughes-Jones Manchester

The EVN-NREN Project

Richard Hughes-JonesThe University of Manchester


Radio Astronomy UDP/IP Streaming 500 Mbit/s iGrid2002 and ER2002 BE & LBE Qos on GÉANT Backbone Web site www.hep.man.ac.uk/~rich/VLBI_web

http://www.hep.man.ac.uk/~rich/VLBI_web


iGRID2002 Radio Astronomy VLBI Demo.

Web based demonstration sending VLBI data A controlled stream of UDP packets 256-500 Mbit/s on the production network Man Amsterdam

32 bit words bits vertical

Bit Stream ~ 7.1 UDP Packets / Data Track Frame

Timestamp Data

Data Track Frame


European Topology: NRNs, Geant, Sites

SuperJANET4

CERN

iGrid 2002

ManchesterJodrell

SURFnet

JIVE


The Works:

n bytes

Wait timetime

Raid0Disc

UDP Data

Raid0Disc

RingBuffer RingBuffer

TCP Control

Web Interface


Receiving Packets

Time

12346

Incoming packet ids

1234

Move 6

Continue with 7

Packets have Header with packet id + Data Put header directly into control area Put data directly into ring buffer No extra copy

Assume UDP packet arrive in order none lost Deposit data into next consecutive slot

Inspect Header Move data onward to correct location Record:

The received inter-packet spacing1-way delay


UDP Throughput on the Production WAN

Manc-UvA SARA 750 Mbit/s SJANET4 + Geant + SURFnet 75% Manchester Access link

Manc-UvA SARA 825 Mbit/s

UDP Man-UvA Gig 19 May 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transmit Time per frame us

Rec

v W

ire

rate

Mb

its/

s

50 bytes

100 bytes

200 bytes

400 bytes

600 bytes

800 bytes

1000 bytes

1200 bytes

1472 bytes

UDP Man-UvA Gig 28 Apr 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transmit Time per frame us

Rec

v W

ire

rate

Mb

its/

s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes


Service

IP transport Can be routed over the production networks

SDH Less overhead cf IP Will need ethernet framing to get into the PC Future high-rate dedicated HW


Traffic Profile

EVN observing (now) 25 day runs 24 hrs/day Future more flexible Telescope regularly switches to a calibration source Same data rate for many hours

Traffic is continuous Ultimate goal: real-time from telescope Tests: sending from disk files


Transport (1): UDP/IP

Demonstrated at iGrid2002 & ER2002 500 Mbit/s 1500 Byte packets 20 µs spacing OK provided packet loss small Backbones Surfnet, SuperJANET and Dante OK (for now)

TCP/IP Basic problem – protocol takes packet loss as congestion signal MAN Ams For BW of1Gbit, if loose 1 frame, takes 94s to run at full

rate

Packet Re-ordering Already seen in demos Normally 1 packet but have seen 2 (or more) at high rate Can cope with this.


Transport (2): Delay / Jitter

Not very important if O 10ms variation Different telescopes have different network delays Will buffer data at the receiver

Single/Multiple Flows Observing rate 1024 Mbit/s = User data rate + telescope overhead + Network overhead

Ethernet frame, IP, UDP, RAST Gigabit NIC wire rate 1 Gbit/s

Use of 2 GigEth interfaces into loacl router then 2.5G SDH


Traffic QoS Classes on GÉANT Backbone

Normal Traffic

Normal Traffic +

Less Than Best Effort 2.0 Gbit/s

Normal Traffic +

Radio Astronomy Data 500 Mbit/s

Normal Traffic +

Radio Astronomy Data +

Less Than Best Effort 2.0 Gbit/s

Max Throughput on 2.5 G PoS


Some Measurements made during ER2002

Manchester – SuperJANET Access link 1 Gbit/s

500 Mbit UDP/IP Incoming depressed



Dante Backbone UK-NL link

Incoming at NL



No LBE

0

2

4

6

8

10

12

14

16

18

20

0 20 40 60 80 100 120 140 160 180 200Transfer number

No.

Out

of

ord

er

0

5000

10000

15000

20000

25000

No

. L

ost

num_badorder

num_lost

With 1.8Gbit LBE

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

0 20 40 60 80 100 120 140 160Transfer number

No.

Out

of

ord

er

0

5000

10000

15000

20000

25000

No

. L

ost

num_badorder

num_lost


Latency Histograms

256 bytes UDP manw-sara0_8nov02

0

5

10

15

20

25

14500 14550 14600 14650 14700 14750 14800 14850 14900

Latency us

N(t

)


0

5

10

15

20

25

30

14500 14550 14600 14650 14700 14750 14800 14850 14900

Latency us

N(t

)

C


0

10

20

30

40

50

14900 14950 15000 15050 15100 15150 15200 15250 15300

Latency us

N(t

)

No LBE 2.0 Gbit LBE UK DE


0

10

20

30

40

50

14900 14950 15000 15050 15100 15150 15200 15250 15300

Latency us

N(t

)


0

20

40

60

80

100

120

15150 15200 15250 15300 15350 15400 15450 15500 15550

Latency us

N(t

)


0

20

40

60

80

100

15150 15200 15250 15300 15350 15400 15450 15500 15550

Latency us

N(t

)


Throughput

manw-sara0_7nov02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Delay between sending frames us

Rec

v W

ire

rate

Mb

its/

s


manw-sara0_LBE2.0G_7nov02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40


Rec

v W

ire

rate

Mb

its/

s


manw-sara0_8nov02

0

100

200

300

400

500

600

700

0 5 10 15 20 25 30 35 40


ou

t o

f o

rder


No LBE 2.0 Gbit LBE UK DE

manw-sara0_LBE2.0G_7nov02

0

500

1000

1500

2000

2500

3000

3500

0 5 10 15 20 25 30 35 40


ou

t o

f o

rder



Packet re-ordering 20 µs Transmit packet spacing 10,000 sent Different behaviour for small packets Still some re-ordering at 40 µs spacing

~100 for 100 bytes 20 for >1000 bytes

0

100

200

300

400

500

600

0 0.5 1 1.5 2 2.5LBE rate Gbit/s

num

out

of

orde

r1472

1200

1000

600

200

100

50

JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard...

Documents

Transcript of JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard...