Post on 19-Mar-2016
description
AstroWise((G)A)VO Meeting 6 May 2004
ASTRO-WISE- federation OmegaCEN
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
Edwin A. Valentijn
AstroWise((G)A)VO Meeting 6 May 2004
((G)A)VO meeting• Goal: Connect raw data to end user in VO
environment– Exchange views on design level– 3 years after– Design – bound / unbound– Review the layers
• Can we build a working near-VO system? – OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO
AstroWise((G)A)VO Meeting 6 May 2004
Programme – 2 rounds• AstroWise
– Vision - Edwin– Implementation – Object maker– Danny– Prototype – demo - Ewout– Object viewer - Michiel
• AstroGrid• AVO• Sextractor - AVO• GAVO
AstroWise((G)A)VO Meeting 6 May 2004
VST
AstroWise((G)A)VO Meeting 6 May 2004
Paranal
Mid 2005
+ 3 year
AstroWise((G)A)VO Meeting 6 May 2004
Lofar
IBM- Blue Gene/L
AstroWise((G)A)VO Meeting 6 May 2004
Large Data Volume• Wide-field imaging instruments, vast amounts of data
– VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data
• Handling of the data is non-trivial– Pipeline data reduction– Calibration and re-calibration– Image comparisons and combinations– Working with source lists– Visualization
• Science can only be archive based
ESOcompliant}
AstroWise((G)A)VO Meeting 6 May 2004
ASTRO-WISE Mission Virtual Survey System
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate dynamically– federated to link different data centers
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
raw pixel data pipelines/cal files catalogues
AstroWise((G)A)VO Meeting 6 May 2004
Keys -Solution
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
AstroWise((G)A)VO Meeting 6 May 2004
Monitoring Photometric Calibration
AstroWise((G)A)VO Meeting 6 May 2004
Calibration procedures
Sanity checks
Quality controlCalibration procedures
Image pipeline
Source pipeline
AstroWise((G)A)VO Meeting 6 May 2004
AstroWise Pipelines
Photometric pipeline
Bias pipeline
Flatfield pipelineImage pipeline
Source pipeline
SourceList
RA0, Dec0
t0
V filterSource
ListV filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
t1
SS
C
I
TES
A
A
O
B filterRA0, Dec0
SourceList
SourceList
V filterRA0, Dec0
SS
C
I
TES
A
A
OS
S
C
I
TES
A
A
O
SourceList
B filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
V filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
B filterRA0, Dec0
SourceListB - VRA0, Dec0
SourceList
V filterRA0, Dec0
t0
SourceList
V filterRA0?, Dec0?
t1
SourceList
V filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
tn
Proper motion
Red quasars
Supernovae
Tbyte source lists brains make the associations
Link -lists
as fast as possible
AstroWise((G)A)VO Meeting 6 May 2004
DB – engine
Oracle contract:– reference – licenses - consultancy
Oracle 9i -> 10g:- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)- Python I/F SQL – OCI- Oracle db- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools– Interoperability: STREAMS connects to others: SyBase, MySQL
AstroWise((G)A)VO Meeting 6 May 2004
Share the loadAstroWise Survey System • Processing
– Hardware • Beowulf processors – 32 (most cases)• Multi Terabyte disks (10 – 100)
– Data reduction• Derive calibration• Run image pipeline (1 Mpx/s)
• Archiving– Storage
• Images (100’s Tbyte), Calibration files (10 Tbyte)• Source parameters (1-10 Tbyte)
– Federate (network speed)• 5 Mb/s (24 hours/day) full replication (MegaCAM 10 Mb/s)• 200 Mb/s no replication, on-the-fly retrieval
AstroWise((G)A)VO Meeting 6 May 2004
Contents of federation• Raw data
– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
AstroWise((G)A)VO Meeting 6 May 2004
Concepts of federation• Federation maintained by a single database- Oracle9i• Full history tracking
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO– Time: Time stamping VO interface
• Software standards– Classes/data model/procedures– 00 – inheritance/ persistency– Python scripts/ c-libraries USER Python
AstroWise((G)A)VO Meeting 6 May 2004
How to use the system?UI: Ingest raw data
UI: Run standard pipelines – GUI
UI: Modify pipeline/modules
UI: Make/associate source lists
5LS: Query – whatever
AstroWise((G)A)VO Meeting 6 May 2004
VST - Virtual Survey Telescope
AstroWise((G)A)VO Meeting 6 May 2004
What ? science ?Collaborate - EU-wide- Projects
- user defined qualifications
- Surveys = projects
- Archive studies = Surveys =projects
AstroWise((G)A)VO Meeting 6 May 2004
New components
• Python binding to Oracle – 5LS ۷• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
AstroWise((G)A)VO Meeting 6 May 2004
Schedule• Hardware
– OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷
– OAC , USM , Terapix + CVS federation ۷• Software
– Design – review Q2 2002- ۷– Design – formalized July 2003 ۷– WFI +INT Prototype operational Q2 2003 ۷– Basic operations – Q4 2003۷– Evaluate and prepare for mass production 2004– Qualify and populate 2005– Deliver survey system – satellites