ASTRO-WISE- federation OmegaCEN
description
Transcript of ASTRO-WISE- federation OmegaCEN
![Page 1: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/1.jpg)
Virtual Survey System sept 04
ASTRO-WISE- federation OmegaCEN
ASTRO-WISE- federation OmegaCEN
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
AstroWise a Virtual Survey System
OmegaCAM – Lofar – AstroGrid –((G)A) VO
Edwin A. ValentijnEdwin A. Valentijn
![Page 2: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/2.jpg)
Virtual Survey System sept 04
VSTVST
![Page 3: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/3.jpg)
Virtual Survey System sept 04
Paranal Paranal
Mid 2005Mid 2005
20072007
![Page 4: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/4.jpg)
Virtual Survey System sept 04
![Page 5: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/5.jpg)
Virtual Survey System sept 04
Large Data VolumeLarge Data Volume
• Wide-field imaging = vast amounts of data– VST sees equivalent of Southern sky with 0.2” pixels in 3 years. 100 Tbyte of image data and Tbytes of source list data A project like KIDS (1000 Sq Deg) has >10^6 8Mpix raw data
images
• Wide-field imaging = vast amounts of data– VST sees equivalent of Southern sky with 0.2” pixels in 3 years. 100 Tbyte of image data and Tbytes of source list data A project like KIDS (1000 Sq Deg) has >10^6 8Mpix raw data
images
• Handling of the data is non-trivial– Pipeline data reduction– Calibration with very limited resources– Things change in time:
–Physical changes (atmosphere, various gains)–Code (new methods, bugs)–Human insight in changes
– Working with source lists
Science can only be archive based
• Handling of the data is non-trivial– Pipeline data reduction– Calibration with very limited resources– Things change in time:
–Physical changes (atmosphere, various gains)–Code (new methods, bugs)–Human insight in changes
– Working with source lists
Science can only be archive based
![Page 6: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/6.jpg)
Virtual Survey System sept 04
ASTRO-WISE Mission Virtual Survey SystemASTRO-WISE Mission Virtual Survey System
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate on the fly dynamically – Paradigm: no static data releases but dynamic on request data– federated to link different data centers
• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate on the fly dynamically – Paradigm: no static data releases but dynamic on request data– federated to link different data centers
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration
raw pixel data pipelines/cal files catalogues raw pixel data pipelines/cal files catalogues
![Page 7: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/7.jpg)
Virtual Survey System sept 04
Keys -SolutionKeys -Solution
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO
• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
• Procedurizing– Data taking at telescope for both science and
calibration data• Observing Modes: —Stare —Jitter —Dither —SSO
• Observing Strategies: —Stan —Deep —Freq —Mosaic
– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and
calibration– View pipeline as an administrative problem
![Page 8: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/8.jpg)
Virtual Survey System sept 04
Calibration proceduresCalibration procedures
Sanity checksSanity checks
Quality controlQuality controlCalibration proceduresCalibration procedures
Image pipelineImage pipeline
Source pipelineSource pipeline
![Page 9: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/9.jpg)
Virtual Survey System sept 04
AstroWise PipelinesAstroWise Pipelines
Photometric pipelinePhotometric pipeline
Bias pipeline
Flatfield pipeline
Image pipeline
Source pipeline
![Page 10: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/10.jpg)
Virtual Survey System sept 04
DB – engineDB – engine
Long search: Objectivity, CERN HEP July 2002 Oracle contract
– reference – licenses - consultancy
Oracle 9i -> 10g:- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)– Interoperability: STREAMS connects to others: SyBase, MySQL– Python I/F SQL – OCI- Oracle db (Python binding)
Long search: Objectivity, CERN HEP July 2002 Oracle contract
– reference – licenses - consultancy
Oracle 9i -> 10g:- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools- federation through “Advanced replication” to evolve into cross
site links (=references= pointers)– Interoperability: STREAMS connects to others: SyBase, MySQL– Python I/F SQL – OCI- Oracle db (Python binding)
![Page 11: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/11.jpg)
Virtual Survey System sept 04
Concepts of federationConcepts of federation
• Federation maintained by a single database- Oracle9i• Full history tracking by linking (joints, references)
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO
– Time: Time stamping VO I/F Publish• Software standards
– Classes/data model/procedures– 00 – inheritance/ persistency
– Python scripts/ c-libraries USER Python
• Federation maintained by a single database- Oracle9i• Full history tracking by linking (joints, references)
– of all input that went into result – providing on-the fly reprocessing
• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO
– Time: Time stamping VO I/F Publish• Software standards
– Classes/data model/procedures– 00 – inheritance/ persistency
– Python scripts/ c-libraries USER Python
![Page 12: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/12.jpg)
Virtual Survey System sept 04
Contents of federationContents of federation
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
• Raw data– Observed images– Ancillary information
• Calibration results– Calibration files time stamped
• Reduced images– Single observation– Coadded images
• Software– Methods (pipelines) for processing calibration– Configuration files
• Source lists – catalogues– Extracted source information– Associated among different data objects
![Page 13: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/13.jpg)
SourceList
RA0, Dec0
t0
V filterSource
ListV filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
t1
SS
C
I
TES
A
A
O
B filterRA0, Dec0
SourceList
SourceList
V filterRA0, Dec0
SS
C
I
TES
A
A
OS
S
C
I
TES
A
A
O
SourceList
B filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
V filterRA0, Dec0
SourceList
B filterRA1, Dec1
SourceList
B filterRA0, Dec0
SourceListB - VRA0, Dec0
SourceList
V filterRA0, Dec0
t0
SourceList
V filterRA0?, Dec0?
t1
SourceList
V filter
t2
RA0?, Dec0?
SourceList
V filterRA0?, Dec0?
tn
Proper motion
Red quasars
Supernovae
Tbyte source lists brains make the associations
Tbyte source lists brains make the associations
Link -lists
as fast as possible
![Page 14: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/14.jpg)
Virtual Survey System sept 04
StatusStatus
• From abstract design to working prototype FIRST VIRTUAL LIGHT
17 April 2003
– [email protected]– INT
• Federated – USM– OAC
• Summer 2004
– First massive ingest
• From abstract design to working prototype FIRST VIRTUAL LIGHT
17 April 2003
– [email protected]– INT
• Federated – USM– OAC
• Summer 2004
– First massive ingest
![Page 15: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/15.jpg)
Virtual Survey System sept 04
VST - Virtual Survey TelescopeVST - Virtual Survey Telescope
![Page 16: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/16.jpg)
Virtual Survey System sept 04
New componentsNew components• Python binding to Oracle – 5LS ۷• Scripting paradigm-۷
– Wrappers– Make metaphor– Persistency, links as attributes
• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
• Python binding to Oracle – 5LS ۷• Scripting paradigm-۷
– Wrappers– Make metaphor– Persistency, links as attributes
• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces
– Tell me everything tool<- >on the fly reprocessing
![Page 17: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/17.jpg)
Virtual Survey System sept 04
New technologies-paradigmNew technologies-paradigm
• raw data centralized – on the fly reprocessing = VSS
• sociology- project management• ++ • Python at top layer(s) = VSS • parallel – IP - seti@home• user tunable pipelines = VSS• ++ db• 5 LS – 5 lines script = VSS• Distribution – VO = VSS
• raw data centralized – on the fly reprocessing = VSS
• sociology- project management• ++ • Python at top layer(s) = VSS • parallel – IP - seti@home• user tunable pipelines = VSS• ++ db• 5 LS – 5 lines script = VSS• Distribution – VO = VSS
![Page 18: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/18.jpg)
Virtual Survey System sept 04
development issuesdevelopment issues
• Indexing Terabytes on the sky• Associating entries • World wide linking entries/distribution
– (referencing) – registries– centralized – replication – p2p– Other p2p’s?
• Visualisation - imaging • Visualisation -multidimensional data• VO world - Euro-VO
• Indexing Terabytes on the sky• Associating entries • World wide linking entries/distribution
– (referencing) – registries– centralized – replication – p2p– Other p2p’s?
• Visualisation - imaging • Visualisation -multidimensional data• VO world - Euro-VO
![Page 19: ASTRO-WISE- federation OmegaCEN](https://reader036.fdocuments.in/reader036/viewer/2022062500/56815721550346895dc4c073/html5/thumbnails/19.jpg)
Virtual Survey System sept 04