ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
1
SD Pipeline Heuristics Status Report
ALMA Project, NAOJ
George KOSUGI
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
2
E2E Pipeline Process Flow
Scheduler
Archive
Offline
Telcal
ASDM (with raw data)
MS2
Filler
Result
Engines
Recipe
ScriptsCurrent Development
•Reference to Science Data•Ref. to Calibrator in a necessary condition
•need to search other calibrator for the sufficient condition?
•Pipeline Parameters
?
Pipeline
Pipeline
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
3
Current SD Pipeline Flow ChartSDFITS
Extract Info
Extract Data
Re-Gridding
FlagSpectrum
Pointing PatternAnalysis
TextTable
TextTable
numarrayPointing,StartTime,IntegrationTime,etc.
BaselineFit
Mapnumarray
Integration over Channels
> Single Pointing> Multi Pointing> Raster/OTF
MS for input data AIPS++Table for Info Storage
MS2MS2
MS for output
FITS
FITS for image
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
4
Environment for SD Heuristics Pipeline
Python Packages– casapy with ASAP– numarray: work area for spectrum data analysis– Scipy: Kmean clustering algorism– Numpy: data for Kmean– PyLab: plotting– PyFits: for collecting info from SDFITS header
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
5
What is necessary for SD UT MS input
– Need sample MS
– MSs for casa/ASAP UT in 2007/03 will be adequate Store/Retrieve info into/from AIPS++Table MS and FITS output
----------------- 2~3 FTE-w Calibration script
– Investigate TelCal Engine
– Investigate scripts used in the casa/ASAP UT
----------------- 2~3 FTE-w ? Additional plots
----------------- 1~2 FTE-w ?
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
6
Baseline Fit is done for each spectrum one by one
Determine Line Free Channels (if not specified)
Fit and Subtract Baseline Calculated from the Line Free Channels
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
8
Determine Line Free Channels (if not specified)
– Subtract DC (bias) and gradient components– Accumulate several spectra in close vicinity in space to e
nhance S/N ratio for line detection– Line Detection by application of ASAP.asaplinefinder– Kmean clustering analysis is applied to the detected lines
(parameters are line center and line width)– Select real (supposed to be) lines by examining continuiti
es in space, distortion of the vector, density of the detection, and cluster size
– Calculate 2-dimensional fit (ra, dec, line center), (ra, dec, line width) to each cluster member
– Apply fit to spectra whose position is within the clustering radius
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
9
Fit and Subtract Baseline Calculated from the Line Free Channels
– Determine fitting order for polynomial• Search gaps in the time sequence• fitting order is determined for each bunch of data (between 2 tim
e gaps)• fitting order is determined from frequencies of major component
s in the Fourier space– Calculate Fit
• to avoid jaggy due to the shortage of the accuracy in the calculation of higher-order power for polynomial, spectrum is divided into sub-spectra, if necessary.
• ASAP.asapfit was used for the fit, however it didn’t converge sometimes. So far python script is used for polynomial fit.
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
10
Spectrum based Flagging (not channel based)
Statistics are measured for each spectrum (emission line channels are excluded):– baseline rms
• before baseline subtraction• after baseline subtraction
– difference from the running mean• before baseline subtraction• after baseline subtraction
– may need other criteria• Tsys, etc...
ALMA Pipeline Heuristics F2F Meeting 2006 in Paris
12
Gridding (Align to Lattice pattern)
Beam size and Grid spacing are given Spectra at positions inside beam radius from th
e grid center are combined into one spectrum– weight: none, gaussian beam, or cone-shaped beam– combine: median (no weight), weighted mean, Nm
axMmin rejected weighted mean
Top Related