Post on 16-Dec-2015
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 1
Data Processing Day
9:00 Data Flow System Deliverables and Integration
10:00 Coffee Break
10:15 The ESO Common Pipeline Library
11:00 The ESO-Reflex Environment
12:00 Lunch break
13:00 CPL Tutorial
15:00 Coffee Break / End of Workshop
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 2
SDD / Pipeline Systems Department
P.Ballester
K.BanseS.CastroL. de BilbaoA. GabaschE. GarciaC. IzzoY. JungJ. LarsenH. LorchL. LundinA. ModiglianiR. PalsaD. PetryK. ShabunJ. Vinther
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 3
Data Flow System Deliverables and Integration
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 4
DFS Deliverables (1): Data Reduction Library
PSD
Paranal Science Operations
INS Commissioning Team
Data Flow Operations
DICB/Archive
Public Release
User Community
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 5
DFS Deliverables (2): ETC/ Observation Preparation Tools
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 6
Talk Outline
• Data Reduction Library• Calibration cascade• Modular design
• Data Products• Science-grade data products• Simulations and validation
• Project Organisation• Project phases and timeline
• DFS Integration• Acceptance tests• DFS infrastructure
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 7
Data Reduction Library
Calibration CascadeModular Design
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 8
Data Reduction Correct for detector and instrument effects Correct for atmospheric effects Separate science data from noise and background
Astronomical Calibrators (Position, Spectral flux, Diameter, etc..)
Instrumental Calibrators(Internal sources)
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 9
Calibration Cascade
Detector Bias
Instrument and DetectorSensitivity
PhotometryStandard Star
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 10
Shift & Add: HAWK-I Modular Design
-Independent modular recipes
-Integrated pipeline recipe
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 12
Shift & Add: Demonstration
1. Raw image (sky = 10,000 star = 1)
2. Dome flat
3. Estimating the sky from N jittered exposures
4. Subtracting the sky and correcting for flat-field
5. Co-adding the sky corrected images
6. Astrometry + Photometry !!
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 13
Science-Grade Data Products
AccuracyRecipes Design
Robustness, Fault toleranceValidation
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 14
Science-Grade Data Products
• Calibrated in physical units with error estimates
• No residual systematic error• S/N close to the achievable optimum (e.g. ETC prediction)
• OB combination
• E.g. fully calibrated and mosaiced images with error bars
• Standard data formats
• SGDPs are independent of the science goals
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 15
Understanding the Instrument Signature (1)UVES Adaptive Optimal Extraction
• UVES archive data reprocessing:• Robust to CCD defects• S/N adaptive optimal extraction• Analytic profile for low S/N• Non-parametric profile for high S/N• Ripple scale QC parameters
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 16
Understanding the Instrument Signature (2)MIDI Visibility Spectra
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 17
Designing Pipeline Recipes
Image Model Optical/Detector model Assumptions on stability/reproducibility
Input Data Signal-to-noise Missing data Contamination Stability (Atm., Ins.) Multiplex/Volume
Reduced Data Random error estimates Systematic errors
Inverse Solution Objective function No assumptions on scientific program Optimization strategy Computational complexity Robustness, Fault tolerance
Pipeline Recipe Default parameters Fixed calibration data
Accuracy
Throughput
Robustness
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 18
Ar lamp - 1” slit - from P. Spano et al. in Merate, July 2007
First light in the lab (VIS arm)
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 19
Using Simulations
Ar lamp - 1” slit - simulated by P. Bristow (ESO)
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 20
State-of-the-art DRS pipeline
Simulated image, multi-pinhole VIS: curved orders, line tilt varying along order
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 21
Robustness, Fault Tolerance
PhysicalModel
3000 4000 5000 6000 7000
30
04
00
50
06
00
Sample Positions on the Sensor
Re
fere
nce
Wa
vele
ng
ths Pattern
Matching
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 22
Project Organisation
SchedulingValidation and Acceptance
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 23
4 Phases
Phase A to
FDR
Documentation/
Prototyping 2-3 years
FDR to
PAE
Development/
Simulation 2-3 years
PAE to
SV and SOP Deployment 6-12 months
Operational Phase Maintenance 10 years
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 24
1618 Document and Technical Documentation
• 1618 Document (currently v.2.0)
• CPL Documentation
• DICB / VLTI DICD
• Gasgano
• Reflex
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 26
DFS Deliverables: 1618 Template Schedule (cont’d)
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 27
From Phase A to FDR
• Phase A and PDR Preparation• Reuse existing documents as templates• Identify extra needs: observation preparation, visualisation, link to the data analysis?
• Learning CPL before the FDR• First recipes in CPL, coding standards, memory management, …• Review existing pipelines, develop prototypes• Coding starts officially only after the FDR
• FDR Preparation• Prototype data reduction algorithms (test and simulated data)• New document: Validation and Test Plan• Observatory Pipeline is template based
— Further processing requirements ?— Data combination, interactive processing
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 28
From FDR to Science Verification
• FDR to PAE• Plan more import validation effort for the first recipes, • In general, plan enough time for testing and finalising• Regular intermediate software releases (3 to 6 months), including test reports• Keep improving simulated or adapted data, in sync with instrument schedule
• PAE • Plan for the complete set of recipes to be ready at PAE• Validated data reduction algorithms (using laboratory and simulated data)• Have ready a few alternative calibration/reduction methods
• Commissioning to Science Verification• Test and validate on instrument and sky data• Identify and solve the unexpected problems
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 29
DICB Approval
• DPR Keyword Values• Ideally, classification rules are only based on DPR keywords• Valid values are listed in the DICB 4.0 documents• Submit proposal to DICB by FDR before using new values
• DICB Validation • No redefinition of existing keywords (database at www.eso.org/dicb)• Keyword names should not be too long• Avoid underscores and special characters
• Multi-HDU files• Always the same structure of data for a given data type• Data extensions come first, auxiliary and optional tables afterward
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 30
Acceptance Tests
• Usage of CPL recipe template
• Following CPL coding standard• Usage of external libraries• Namespace protection
• Execution Tests• Completeness of the set of recipes
and DRL functions• Availability/representativity of test
data• Proper execution of recipes• Generation of products• Memory leaks• Unit tests• Documentation
• Detailed Validation• Correctness of results• Validation of input• FITS compliance• User-friendly documentation• Data reduction cascade• Unit tests
• Performance and Portability
• Execution speed• Standard platforms: Scientific
Linux
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 31
DFS Integration
Observatory Pipeline Quality Control Pipeline
Desktop science-grade data reductionMaintenance
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 32
Data reduction environments
• Observatory Pipeline • On-the-fly data processing (event driven)• Template-based processing• Static calibration database (only certified products are used)
• Quality Control Pipeline• Batch processing of complete data sets (all science and calibration data produced by one
ESO instrument in one night)• Best available calibrations are used => data must be organized according to the
Calibration Cascade
• Desktop science-grade data reduction• Modular and additional recipes are avilable• Several front-ends for scripting (esorex), browsing (Gasgano), interactive (Reflex)
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 33
Observatory Pipeline
Instrument Instrument Raw DataRaw Data
ArchiveArchive
PIPELINEPIPELINE OFFLINEOFFLINE
FurtherFurtherAnalysisAnalysis
Raw data
Processed data
On-Line Archive SytemOn-Line Archive Sytem
ShippingShipping
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 34
Observatory Pipeline: Workstation
Pipeline workstation
Data Organizer
Reduction Block Scheduler
Configuration files
Calibration database
Data Reduction SystemPRODUCTS
Instrument
Package
Common Pipeline LibraryArchive
Quality control
Data arriving from the instrument
Reduction Block List of the raw frames
List of the calibration data
Name of the products
DRS recipe to apply
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 35
OCA Rules
if DPR.CATG=="CALIB" and DPR.TYPE=="LAMP,WAVE" then{ DO.CLASS = "ARC_SPECTRUM"; RAW.TYPE = "WAVE";}
1. CLASSIFICATION
2. ORGANIZATION
select execute(GI_WAVE_CALIBRATION) from inputFiles where RAW.TYPE=="WAVE“ group by TPL.START
3. ASSOCIATION
action GI_WAVE_CALIBRATION{select file as MASTER_BIAS from calibFiles wherePRO.CATG=="MASTER_BIAS" and inputFile.DET.WIN1.BINX==DET.WIN1.BINX;
select file as GRATING_DATA from calibFiles wherePRO.CATG=="GRATING_DATA" and inputFile.INS.GRAT.NAME==INS.GRAT.NAME;
recipe giwavecalibration { }
product mflat { PRO.CATG="MASTER_FLAT"; }}
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 36
Quality Control Log Files
12:21:51>-START GROUP / Start [AMBER]
12:21:51> ARCFILE = 'AMBER.2007-03-21T12:16:08.099.fits' [AMBER]
12:21:51> TELESCOP = 'NOT_SPECIFIED' / Telescope [AMBER]
12:21:51> INSTRUME = 'AMBER' / Instrument name [AMBER]
12:21:51> OBSERVER = 'UNKNOWN' / Observer name [AMBER]
12:21:51> PIPEFILE = 'p2vm.fits' / Filename of data product [AMBER]
12:21:51> INS GRAT1 NAME = 'GHR' / Grating common name. [AMBER]
12:21:51> INS GRAT1 RESOL = 527.778; / Encoder resolution [Enc/deg]. [AMBER]
12:21:51> INS GRAT1 WLEN = 2364.972; / Grating central wavelength [nm]. [AMBER]
12:21:51> INS GRAT1 ZORDER = 40319; / Grating zero order position [Enc]. [AMBER]
12:21:51> INS GRIS1 NAME = 'NAR_SLT' / OPTIi name. [AMBER]
12:21:51> INS GRIS2 NAME = '3T_K' / OPTIi name. [AMBER]
12:21:51> INS MODE = '3Tstd_High_K_1_2.365' / Instrument mode used. [AMBER]
12:21:51> PRO DID = 'ESO-VLT-DIC.PRO-1.15' / Data dictionary for PRO [AMBER]
12:21:51> PRO CATG = 'P2VM_REDUCED' / pipeline product category [AMBER]
12:21:51> PRO TYPE = 'REDUCED' / Product type [AMBER]
12:21:51> PRO REC1 ID = 'amber_p2vm' / Pipeline recipe (unique) identifier [AMBER]
12:21:51> PRO REC1 DRS ID = 'cpl-3.0' / Data Reduction System identifier [AMBER]
12:21:51> PRO REC1 PIPE ID = 'AMBER/2.3.2' / Pipeline (unique) identifier [AMBER]
12:21:51> PRO REC1 RAW1 NAME = 'AMBER.2007-03-21T12:13:38.479.fits' / File name of raw frame [AMBER]
12:21:51> PRO REC1 RAW1 CATG = 'AMBER_3WAVE' / Frame category of raw frame [AMBER]
12:21:51> PRO DATANCOM = 14; / Number of frames combined [AMBER]
12:21:51> PRO REC1 CAL1 NAME = 'FlatFieldMap.fits' / File name of calibration frame [AMBER]
12:21:51> PRO REC1 CAL1 CATG = 'AMBER_FLATFIELD' / Frame category of calibration frame [AMBER]
12:21:51> PRO REC1 PARAM1 NAME = 'dummy' / Name of recipe parameter [AMBER]
12:21:51> DET NTEL = 3; / Number of telescopes [AMBER]
12:21:51> QC P1 OFFSETY = 0.02; / Offset wavelength calibration [AMBER]
12:21:51> QC P2 OFFSETY = 0.05; / Offset wavelength calibration [AMBER]
12:21:51> QC P3 OFFSETY = 0.03; / Offset wavelength calibration [AMBER]
12:21:51>-STOP GROUP / Stop [AMBER]
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 37
Pipeline @ Garching: QC
•Main Tasks• Create master calibrations• Derive and trend QC parameters• Create science products (Service Mode)• Prepare data packages (Service Mode)• Pipelines & QC parameters: requirements & testing
•Customers• Paranal Science Operations• ESO community (PIs, archive users)
Home Page: www.eso.org/qc
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 38
Information by Instrument
Similar pages for every VLT/VLTI instrument
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 39
Public Releases
• www.eso.org/pipelines
• Linked from all ESO instrument pages
• Release package, documentation, demonstration data
13
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 40
Desktop Data Reduction: Gasgano
•VLT interactive data organisation toolFITS file browsingGroupingClassification
•Interactive front-endInterface to CPL pluginsInterface to vizualisation tools
•FeaturesJava languageFITS format
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 41
Desktop Data Reduction: Reflex
- CPL recipes
- External tools- Python scripts- Visualisation
- Beta users and internal evaluation
SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 42
Pipeline Maintenance
• DFS Tickets• Issued by PSO, DFO, INS, and the user community
• Pipeline priority meetings• Bi-yearly meetings with representation of INS, DFO, PSO, SDD• General issues and for each instrument closed, in process, open tickets• Priority setting for further pipeline development
• Instrument Evolution• Commissioning of new modes• Detector upgrades