GSB : A GSB : A real-timereal-time SSoftware oftware

bback-end for the ack-end for the GGMRTMRT

Jayanta Roy

National Centre for Radio Astrophysics

Pune, India

LFRU@GMRT, 12th December 2008

Collaborators : Collaborators :

Yashwant Gupta (NCRA)Yashwant Gupta (NCRA)

Ue-Li Pen (CITA)Ue-Li Pen (CITA)

Jeff Peterson (CMU)Jeff Peterson (CMU)

Jitendra Kodilkar (NCRA)Jitendra Kodilkar (NCRA)

Sanjay Kudale (NCRA)Sanjay Kudale (NCRA)

AcknowledgmentsAcknowledgments

Analog group of GMRT : Ajitkumar and his group designed a new full bandwidth

base-band receiver for GSB.

Computer management group of GMRT : Mangesh Umbarje and Sumit Mirajkar.

Mechanical support : By H S Kale and his group.

Real-time phase solution from Santaji Katore.

Clock and trigger distribution unit from Kevin Bandura (CMU)

Discussion and suggestions from other NCRA staff members :

Jayaram Chengalur, Subhashis Roy, Rajaram Nityananda, Vasant Kulkarni

Desired operating modesDesired operating modes

32 antennae x 2-pols 32 MHz real-time software correlator

high time resolution incoherent and coherent

array beams formation

(GSB is aimed to provide the specs of existing backend + more)

32 antennae x 2-pols 32 MHz base-band recording by real-time streaming to

disk array with off-line read-back of recorded data and computation

Input data rate : 32 antennae x 2-pols base-band analog inputs @ 16/32 MHz of bandwidth

--> 2 GSamples/sec (using 16 ADC cards with 4 analog inputs in each card)

Implementation detailsImplementation details

Basic Methodology : Basic Methodology :

Run synchronous sampling on all 16 ADC boards – 16/32 MHz BW

Transfer data from ADC board to CPU unit (via interrupt driven DMA @ 145 MB/s)

in large blocks

For baseband recording mode, synchronous write to disk at each recording node

For correlations, distribute data from all antennas (using time division multiplexing)

to all compute nodes -- each node handles 1/N time slice from each block

Carry out FFT, post-FFT fringe stop, Correlation and array beam formation at each

compute node

Record integrated visibilities results and beam data to the “collector nodes”

The performance-per-watt

capability of the GSB cluster

~ 260 Mflops/watt

System PerformanceSystem Performance

Correlator specs : Spatial resolution = 32kHz – 125kHz

Temporal resolution = 1s – 16s

Max output data rate = 14 GB/hr

Beamformer : Incoherent/Coherent intensity mode : Temporal resolution = 60 us + Max output data rate = 57 GB/hr

Coherent array voltage data : Temporal resolution = nyquist limited

Max output data rate = 200 GB/hr

Base-band recorder : Temporal resolution = nyquist limited

Max output data rate = 3.5 TB/hr

GSB with ONLINE Control and Monitor interfaceGSB with ONLINE Control and Monitor interface

GSBMON : GSB real-time data Monitoring GSBMON : GSB real-time data Monitoring

GSB current statusGSB current status

32 antennae x 2-pols 16 MHz real-time software correlator running regularly

real-time array phasing

formation of high time resolution incoherent and

coherent array beam

32 antennae x 1-pols 32 MHz real-time software correlator being tested

32 antennae x 2-pols 16 MHz 4-bit base-band recording by real-time streaming to disk

array with off-line read-back of recorded data and computation

Some results from GSB .......Some results from GSB .......

Phases from GHB Phases from GSB

Preliminary results from the GSBPreliminary results from the GSB

GSB gives ~ 1.5 times better rms than GHBGSB gives ~ 1.5 times better rms than GHBGSB Image of NGC 4631 (made by J. Kodilkar)

GSB image over-plotted with data from Palmor observatory sky surveyGSB image over-plotted with data from Palmor observatory sky survey

Image made by J. Kodilkar

Single pulses Single pulses

time series of time series of

PSR B0329+54PSR B0329+54

GSB pulsar observationsGSB pulsar observations

Pre detection dispersion removalPre detection dispersion removal““Presto” folded profile of B0950+08 Presto” folded profile of B0950+08

Comparative study of GSB and DiFx pipelineComparative study of GSB and DiFx pipeline

Input is same baseband recorded Input is same baseband recorded

data from GSB baseband recorderdata from GSB baseband recorder

GSBC and DiFx produce similar GSBC and DiFx produce similar

results, but GSBC runs significantly results, but GSBC runs significantly

faster than DiFx ! faster than DiFx !

Auto spectra Auto spectra Cross spectraCross spectra

Spectral variation of RMSSpectral variation of RMS

DiFx is Swinburne’s VLBI

software correlator (Deller et al)

Enhanced Performance from GSB : Pre and post detection RFI removalRFI removal

Power-line RFI removal

Time domain flagging on raw voltage samples

Spectral domain filtering of powerline harmonics in the visibility data

Time domain RFI removal of power-line and any impulsive RFI

Median absolute deviation (MAD) based estimator

SummarySummary

Real-time GSB is a highly optimized multi-threaded vectorized parallel pipeline,

working ~ 90% of the theoretical peak FLOPS.

GSB is a fully real-time software back-end: First instance of such a real-time

back-end for an intermediate size multi-element interferometer array like

GMRT.

GSB is going to be the 24 x 7 default observatory back-end.

Radio frequency Interference: Severe at low frequencies; GSB processing

algorithms are demonstrated to handle these issues.

GSB development cycle is only of 3 yrs.

Next steps Next steps

High BW real-time software back-end : GSB is easily reconfigurable

Implement 4-bit 200 MHz BW 8 antennae back-end.

Implement 4-bit 100 MHz BW 16 antennae back-end

GPU based high intensive pulsar processing : coherent dedispersion in real-time

and RFI removal in real-time

Formation of Multiple phased array beams within the primary beam

24 x 7 piggy-back fast transient survey pipeline on GSB

Software based processing likely to be a future growth path of radio astronomy signal Software based processing likely to be a future growth path of radio astronomy signal

processing : GSB can be a test bench for thatprocessing : GSB can be a test bench for that

Multiple phased array beam formationMultiple phased array beam formation

FFT cost/ node ~ 5 GFLOPS Beam-forming (# 400) cost/ node ~ 160

GFLOPS

1.6 x of real-time on 72 nodes itanium cluster

Compute requirements :Compute requirements :

Real-tim

e space

GMRT Multibeaming ……GMRT Multibeaming ……

Multi-element arrays can shape the beams to produce Multi-element arrays can shape the beams to produce nulls in directions of known,nulls in directions of known,

spatially fixed sources of RFIspatially fixed sources of RFI

Signals from multiple beams (e.g. one beam on-target, another off-target) can beSignals from multiple beams (e.g. one beam on-target, another off-target) can be

useful for implementing useful for implementing adaptive RFI cancellationadaptive RFI cancellation techniques techniques

Desirable to have a Desirable to have a on-line calibrationon-line calibration by having one beam continuously on the by having one beam continuously on the

in-field calibratorin-field calibrator

Sensitivity and computational load balancingSensitivity and computational load balancing : need to synthesize ~ 500 multiple : need to synthesize ~ 500 multiple

phased array beams, pointing in different directions within primary beam.phased array beams, pointing in different directions within primary beam.

synthesize partially synthesize partially dephased fatter beamsdephased fatter beams of lower sensitivity : reducing the of lower sensitivity : reducing the

number of beams number of beams

24 x 7 Transient search using GSB : 24 x 7 Transient search using GSB :

Study of dynamic radio skyStudy of dynamic radio sky

– Collaborative effort of NCRA and Swinburne University of Technology– Running in piggy-back mode simultaneously with other observations– Object of interest : Fast transients : nanosecond to 100's of millisecond– Event detection : based on the sensitivity of 8 antennae incoherent array

beam

over 32 MHz

Phase space of the survey

Coincidence detections in all 4 incoherent beams : Discrimination real Coincidence detections in all 4 incoherent beams : Discrimination real

signals from RFIsignals from RFI

Search in Dispersion measure space : Discriminate out fast radio transient from RFI.

Coincidence or anti-coincidence filter : Multiple sub-array multiple beam

coincidence filter reduces the false triggers due to direction-location dependent RFI.

Efficient RFI removal technique : MAD based technique (as described before.....)

…….. Discrimination of real signals from RFI.. Discrimination of real signals from RFI

24 x 7 Transient search pipeline24 x 7 Transient search pipeline

SummarySummary

GSB + Itanium cluster with its compute and storage capability will open

up the phase space of GMRT to enhance its productivity.

GSB is going to be the 24 x 7 default observatory back-end.

Total GSB development cycle (from first idea to complete realisation)

has been only 3 yrs.

Thank youThank you