e-VLBI: Overview and Update

Alan R. Whitney

MIT Haystack Observatory

Traditional VLBIThe Very-Long Baseline Interferometry (VLBI) Technique(with traditional data recording)

The Global VLBI Array(up to ~20 stations can be used simultaneously)

Chautauqua 2001

Quasars, hotspots, polarization

VLBI astronomy example

ASTRONOMY• Highest resolution technique available to

astronomers – tens of microarcseconds• Allows detailed studies of the most distant

objects

GEODESY• Highest precision (few mm) technique

available for global tectonic measurements• Highest spatial and time resolution of

Earth’s motion in space for the study of Earth’s interior

•Earth-rotation measurements important for military/civilian navigation•Fundamental calibration for GPS constellation within Celestial Ref Frame

VLBI Science

Plate-tectonic motions from VLBI measurements

• VLBI data still largely recorded on magnetic tape

• Tapes are ~$1000 each and are no longer available

• Shipping costs are high and delays can be weeks

How most VLBI is done today

Mark 4 VLBI Correlator at Haystack Observatory

• Correlator unit can simultaneously process up to 16 stations at 1Gbps/station

Scientific Advantages of e-VLBI

• Bandwidth growth potential for higher sensitivity– VLBI sensitivity (SNR) proportional to square root of Bandwidth

resulting in a large increase in number of observable objects(only alternative is bigger antennas – hugely expensive)

– e-VLBI bandwidth potential growth far exceeds recording capability(practical recordable data rate limited to ~1 Gbps)

• Rapid processing turnaround– Astronomy

• Ability to study transient phenomena with feedback to steer observations

– Geodesy• Higher-precision measurements for geophysical investigations

• Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation

Practical Advantages of ‘e-VLBI’

• Increased Reliability– remove recording equipment out of field

– remote performance monitor & control capability in near real-time

• Lower Cost– Automated Operation Possible

• eliminates manual handling and shipping of storage media

– Near Real-time Processing• forestalls growth of storage-capacity requirements with bandwidth growth

– Elimination of recording-media pool (millions of $’s!)

• Avoid unexpected media-shipping interruptions and losses

Elements of e-VLBI Development

• Phase 1: Develop eVLBI-compatible data system– Mark 5 system development at MIT Haystack Observatory being supported by NRAO, NASA,

USNO plus four international partners

– Prototypes now deployed in U.S. and Europe

• Phase 2: Demonstrate 1 Gbps e-VLBI using Bossnet(w/ DARPA and NASA support)

– ~700km link between Haystack Observatory and NASA/GSFC

– First e-VLBI experiment achieved ~788Mbps transfer rate

• Phase 3: Establish adaptive network protocol(newly awarded NSF grant to Haystack Observatory; collaboration with MIT Lab for Computer Science and MIT Lincoln Laboratory);

– New IP-based protocol tailored to operate in shared-network ‘background’ to efficiently using available bandwidth

• Phase 4: Extend to global scale

– High-speed global research networks will facilitate the global application of e-VLBI

Mark 5 VLBI Disk-Based Data System (Phase 1)

• 1 Gbps continuous recording/playback to/from set of 8 inexpensive (ATA) disks• Developed at MIT Haystack Observatory with multi-institutional support• Mostly COTS components• Two removable ‘8-pack’ disk modules in single 5U chassis• With currently available 200GB disks – capacity of single ‘8-pack’ 1.6TB; expected to increase to 2.5TB by early 2003 at cost of ~$1/GB• GigE connection for real-time and quasi-real-time e-VLBI operations• Inexpensive: <$20K• ~20 Mark 5 systems now installed at stations and correlators

Haystack

Westford

NASA/GSFCUSNO

(correlator)

(correlator)

Bossnet 1 Gbps e-VLBI demonstration experiment(Phase 2)

Initial experiment

Future

WS/Mark 5

Summit1i

Summit5i

Westford Antenna Millstone RPE Bldg (Lorraine)

Haystack Correlator

GigEGigE

GigE GigESummit

5i

GigE

Congress StBoston

GlowNet

Lincoln Lab (Lorraine/Steve)

A1

A2

B5

B1

D2

Cisco1500

A4

C3B2

B4

C1

B3

D3

Cisco1500

Summit5i

Summit5i

Amp

E1

E3

Bossnet(Steve/Jim) ISI-E

E2 E6

A3

Figure 1: e-VLBI Path - Haystack to ISI-E

C2

GigEB6

WDM

WDM

GigE GigE

F2

Amp Amp

GlowNetD1

E8

F1

GigE

WDM

evlbi011.drw20 Dec 01

WS/Mark 5

WS/Mark 5

1 8

14

13

16

15

28-1A

28-1B

LL

(Shared with LL onscheduled basis)

To be upgraded from2 to 3

To be replaced byJuniper M20 Jan 02 with OC-48 to Bossnet

1310

TestWS

To be providedby Haystack

MITCampus

1510

Plan install by 1 Jan 02

E4

E7

E5

GigE

Other users(no control)

GigE

To be updatedto OC48 Jan02

Scheduling onBossnet calendar

Schedule to becoordinated with LL

Schedule to becoordinated with LL

Dedicated -no scheduling

Dedicated -no scheduling

Details of path from Haystack to ISI-E – work in progress!

Trans-

ponder

ISI-E (Tom)

G1

Figure 2: e-VLBI Path - ISI-E to GSFC/GGAO

JuniperM40

JuniperM160

JuniperM160

Cisco3508

Switch

Summit5i

WS/Mark 5

UMCP (Jerry or Dan)

GSFC/Bldg 28 (Pat) GGAO/Bldg 202 (Pat) GGAO Antenna Trailer (Bill)

WDM

WDM

WDM GigEGigEGigEGigE

OC-48Bossnet

G3

G5

G10 H3

H7

J3

J5

K1

K2

K3

K4

K7 L1

L2

L3

L4

evlbi011.drw20 Dec 01

G2

G4 G8

H8

H9

J4

H7

OC-48

GigEGigE

GigEGigE

Max @ ISI-E (Jerry)

Borrowed OC48interface - may notbe able to keep

Switch

GigEGigEBackup plan

Preferred e-VLBI path,but may not be in place.Pat working with Jerry and Dan.

Other users ( no control)

No jumbo frames

Max @ ISI-E (Jerry)

LuxN

Summit5i

TunerGigE GigE

OC-48

Plan to update toOC-48 ~Jan 02

Summit5i

On orderMAC on orderdue Jan-Feb 02

Dell Linux currently at Haystack willalso be available here

1600 ft of fiberto be installedJan 02

Other users(no control)

1310

G9

G11

G12 H1TestWS?

G6

G7

TestWSMac with

Yellowdog Linux;need details

K5

K6

J1

GigE

Plan mid Jan 02

J2

Need details

Eventually replacedwith Mark 5 unit

Need details

H6

TestWS?

H5 Need details

TestWS?

H4

Scheduling onBossnet calendar

No scheduling;sufficient capacityfor e-VLBI

Scheduling necessaryif this path used

Will try to scheduleeither K2 or K3exclusively fore-VLBI experiment

Dedicated - noscheduling necessary

H2

J6

Details of path from ISI-E to GSFC Antenna – work in progress

Performance test results – Haystack/GGAO

Average sustained rate >900 Mbps over 10 hours

Westford-GGAO e-VLBI results

• First near-real-time e-VLBI experiment conducted on 6 Oct 02– Recorded data at 1152 Mbps on Westford-GGAO baseline– GGAO disk-to-disk transfer at average 788 Mbps transfer rate– Immediate correlation on Haystack Mark 4 correlator– Nominal fringes observed

• Direct data transfer experiment conducted on 24 Oct 02– Direct transfer of GGAO data to disk at Haystack at 256 Mbps– Immediate correlation with Westford data– Nominal fringes observed

• Next step – full real-time e-vlbi– Mark 5 system is capable of transmitting in real-time– But, still need additional work on correlator software to synchronize

correlator operation to real-time– Hope to conduct first experiment in early 2003

• Conclusion– e-VLBI at near Gbps speeds over ordinary shared networks is possible

but still difficult

Preliminary international e-VLBI experiments

• Westford, MA to Kashima, Japan - experiments in Oct 02 and Mar 03– Files exchanged over Abilene/GEMnet networks

• Nominal speed expected to be ~20 Mbps, but achieved <2 Mbps for unknown reasons - investigating

– Correlation on Mark 4 correlator at Haystack and PC Software correlator at Kashima; nominal fringes obtained

– Further experiments are scheduled; network tuning is in progress

• Kauai, Hawaii to Wettzell, Germany (in progress)– Daily experiments of ~100GB are ideal candidate for early e-VLBI

– Data will be transferred to Haystack Observatory for processing(OC-3 speeds are possible)

– Network links are now being brought up

New IP Protocols for e-VLBI (Phase 3)

• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity

• New protocols are being developed to utilize networks in ‘background’ mode for applications such as e-VLBI

– Take advantage of special characteristics of e-VLBI data

– Will ‘scavenge’ and use ‘secondary’ bandwidth

– Will give priority to ‘normal’ users

– Requires a new ‘end-point adaptive strategy’

• Work being carried out by MIT Haystack Observatory in collaboration with MIT Laboratory for Computer Science and MIT Lincoln Laboratory

– 3-year program; will demonstrate e-VLBI connections both nationally and internationally

– Network researcher Dr. David Lapsley hired to lead this effort

Typical bit-rate statistics on Abilene network

Conclusion: Average network usage is only a few % of capacity

Usage >20Mbps less than 1% of the time

100 500 Mbps

0.1

1.0

0.01

0.001

Typical distribution of heavy traffic on Abilene

Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes

secs200 400 1000

1.0

0.9

0.8

0.7

<10% of ‘bulk’ transfers exceed ~100 secs

Further extensions to national and global community(Phase 4)

• Possibilities for international connections– Surfnet – U.S. to Europe at 2.5 Gbps – TransPAC – U.S. to Japan at 655 Mbps (upgrade to 1.2 Gbps planned)– GEMnet – currently ~20 Mbps, planning to upgrade to 2.2 Gbps– APAN – U.S. to Japan at 655 Mbps– AMPATH – possible connections to telescopes in Chile and Brazil– A sample of others under construction

• TEIN – Paris to Seoul• EUMEDCONNECT – Europe to Mediterranean• NeDAP – Europe to Russia• ALIS – Europe to Latin and South America

– One of the most interesting!• IEEAF – new Europe/U.S. link at 10 Gbps! • Donated by Tyco, Inc.

– Dedicated-lambda networks are being investigated as possibility

622 Mbps +10 Gbps

Transoceanic donations to IEEAF (in red)

Northern Europe donations to IEEAF (in red)

TransPAC Network

AMPATH: Research and Education Network and International Exchange Point for the Americas

Launched in March 2000 as a project led by Florida International University (FIU), with industry support from Global Crossing (GX), Cisco Systems, Lucent Technologies, Juniper Networks and Terremark Worldwide

Enables wide-bandwidth digital communications between the Abilene network and 10 National Research and Education Networks (NRNs) in South and Central America, the Caribbean and Mexico

Provides connectivity to US research programs in the region

AMPATH is a project of FIU and the National Science Foundation’s Advanced Networking Infrastructure & Research (ANIR) Division

Note: VLBI telescopes currently in Chile and Brazil

Summary - Impact of e-VLBI Program

• e-VLBI opens new doors for astronomical and geophysical research.

• e-VLBLI represents an excellent match between modern Information Technology and a real science need.

• e-VLBI motivates the development of a new shared-network protocol that will benefit other similar applications.

• e-VLBI drives an innovative IT research application and fosters a strong international science collaboration.