The Astro-wise pipeline

PythonData processing - the low-level interfacesData administration - persistent objects

Python

● Object Oriented scripting language● clean, easy-to-understand syntax● extensive library ● powerful built-in data types (str, dict, list, file)● byte-code interpreted, dynamically typed

Rapid development and easy maintenance

Python, a 1-slide course

wordcount = {}

for line in file('my_thesis.txt'):

for word in line.split():

if word not in wordcount:

wordcount[word] = 1

else:

wordcount[word] += 1

print wordcount

Image processing

● eclipse libraray in C (ESO, N. Devillard)● wrapped with SWIG● fits images and headers as built-in data types >>> from eclipse import image, header

>>> img = image('science.fits') >>> hdr = header('science.fits') >>> flat = image('flat.fits') >>> img = img / flat >>> hdr['FLATFLD'] = 'flat.fits' >>> img.save('flatfielded.fits', hdr)

● Co-addition with SWARP (IAP, E. Bertin)

Catalog Processing

● Sextractor (IAP, E. Bertin)● LDAC (OmegaCAM, E. Deul)

– Astrometry– Photometry

● Common interfaces, but no Python types● Persistent Sourcelists (see KGB)

Astro-wise Pipelines

● Pipelines are about data-administration, not about data-processing

● Persistent objects

– Object's state persists across program boundaries– Python classes representing SQL tables– Should also work without back-end

● Distinguish meta-data and bulk (FITS) data

– meta data in persistent objects– bulk (FITS) data through file-server

The make metaphor

● 'Making' objects

– think unix Makefile– targets and dependencies (recursive)

>>> bias = BiasFrame() >>> bias.raw_frames = [RawBiasFrame('bias1.fits'), RawBiasFrame('bias2.fits'), ... RawBiasFrane('biasN.fits')] >>> bias.make()

An example with Queries

● Queries are Python expressions (see Danny)● Dependencies can be filled through queries

>>> bias = BiasFrame() >>> query = ((RawBiasFrames.chip.name == 'ccd50') & (RawBiasFrames.DATE_OBS > a_date-1) & (RawBiasFrames.DATE_OBS < a_date+1)) >>> bias.raw_frames = list(query) >>> bias.make() >>> bias.store() # the bulk FITS data >>> bias.commit() # the meta data

Image Pipeline