AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5...
-
Upload
lindsey-lucas -
Category
Documents
-
view
216 -
download
1
Transcript of AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5...
![Page 1: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/1.jpg)
AstroGrid: The UK’s Virtual Observatory
Dr Dugan Witherick – Astrophysics Group, UCL
Wednesday 5th December 2007The University of Warwick
![Page 2: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/2.jpg)
AstroGrid – The UK’s Virtual Observatory
Outline of this Seminar
• Describe the different types and volume of data that astronomers use
• Discuss the concept of Virtual Observatories and how they might solve some of these problems
• Introduce AstroGrid as the UK’s Virtual Observatory
![Page 3: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/3.jpg)
AstroGrid – The UK’s Virtual Observatory
The Types and Volume of Data in Astronomy
![Page 4: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/4.jpg)
AstroGrid – The UK’s Virtual Observatory
Ground Based Observatories
Copyright RASC Calgary Centre
![Page 5: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/5.jpg)
AstroGrid – The UK’s Virtual Observatory
Space Observatories
Copyright NASA
Copyright ESA
Copyright ESA/D. Ducros
![Page 6: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/6.jpg)
AstroGrid – The UK’s Virtual Observatory
Data Products Provided by Observatories
For many years astronomers have been provided the raw data products and were required to reduce the data to useable forms themselves. Increasingly, images and spectra are provided in a reduced form as they are reduced automatically through a pipeline.
![Page 7: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/7.jpg)
AstroGrid – The UK’s Virtual Observatory
Astronomical Surveys
• Sloan Digital Sky Survey or SDSS– Aim to map 25% of the sky– Image (through 5 coloured filters) around 100 million
objects– Take spectra of around 1 million objects
• Dark Energy Survey or DES– Plan to collect data on 300 million galaxies– 520 Megapixel camera installed on the existing 4-metre
Blanco Telescope in Chile– Survey to begin in 2009
![Page 8: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/8.jpg)
AstroGrid – The UK’s Virtual Observatory
How many bytes?• Multimission Archive
– Nearly 50 Tb, over half is from the HST (Aug 2007)– Approx. 24 Tb served from Jan-Aug 2007
• European Southern Observatory Archive– Approx. 65 Tb (Aug 2007)– Over 12 Tb (public data) served every year
• Sloan Digitized Sky Survey– 40 Tb when complete
• Visible and Infra-red Survey Telescope for Astronomy (VISTA)– 315 Gb per typical night, est. 150 Tb per year
![Page 9: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/9.jpg)
AstroGrid – The UK’s Virtual Observatory
How many astronomers are accessing these data?
• American Astronomical Society– Approximately 6500 members (source AAS
Nov 2007)
• Royal Astronomical Society– Approximately 3000 members (source RAS
Nov 2007)
![Page 10: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/10.jpg)
AstroGrid – The UK’s Virtual Observatory
Accessing Data
• Digital Audio Tape (DAT) and Linear-Tape Open (LTO)
• CDs, DVDs and increasingly Blu-Ray and HD DVD
• USB Flash Drives/Sticks
• Online– Common Gateway Interface (CGI)– Web Services– HTML Forms
![Page 11: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/11.jpg)
AstroGrid – The UK’s Virtual Observatory
Online Access - Multimission Archive
![Page 12: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/12.jpg)
AstroGrid – The UK’s Virtual Observatory
Online Archive - ESO Archive
![Page 13: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/13.jpg)
AstroGrid – The UK’s Virtual Observatory
Online Access - SDSS Skyserver
![Page 14: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/14.jpg)
AstroGrid – The UK’s Virtual Observatory
Virtual Observatories and how they can help
![Page 15: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/15.jpg)
AstroGrid – The UK’s Virtual Observatory
What are Virtual Observatories (VOs)?
“The VO vision can be summed up as the desire to make all archives speak the same language – all searchable and analyzable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.”
September 2003
Professor Andy Lawrence FRSEInstitute for AstronomyRoyal Observatory Edinburgh
![Page 16: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/16.jpg)
AstroGrid – The UK’s Virtual Observatory
The Virtual Observatory Aim
• The astronomer should be able to easily search all astronomical data worldwide from anywhere– A single user interface which can find/search all
available data
• It should be possible to view and analyse the data using a common set of tools– The returned data products should use common
formats
![Page 17: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/17.jpg)
AstroGrid – The UK’s Virtual Observatory
Creating a Virtual Observatory - An International Effort
The International Virtual Observatory Alliance
"facilitate the international coordination andcollaboration necessary for the development anddeployment of the tools, systems and organizationalstructures necessary to enable the internationalutilization of astronomical archives as an integrated
and interoperating virtual observatory."
![Page 18: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/18.jpg)
AstroGrid – The UK’s Virtual Observatory
As a result of the IVOA we have
• A common Resource Metadata
• A common data querying language– Astronomical Data Query Language (ADQL)– Based on Structured Query Language (SQL)
• Common table format– Virtual Observatory Table (.vot)
![Page 19: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/19.jpg)
AstroGrid – The UK’s Virtual Observatory
Identifying these resources
• International Virtual Observatory Resource Name or IVORN– Very similar to a Uniform Resource Location (URL)– Ivo://community.authority.id/resource/etc…
• Example 1 - ivo://nasa.heasarc/hstpaec– HST planned and archived exposures catalogue– Provided by NASA’s High Energy Astrophysics Science
Archive Research Center
• Example 2 - ivo://wfau.roe.ac.uk/sdssdr5-dsa– SDSS Data Release 5– Provided by Wide Field Astronomy Unit at the Royal
Observatory, Edinburgh
![Page 20: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/20.jpg)
AstroGrid – The UK’s Virtual Observatory
AstroGrid
![Page 21: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/21.jpg)
AstroGrid – The UK’s Virtual Observatory
What is AstroGrid?
AstroGrid is a UK government funded, open source project
designed to create a working Virtual Observatory (VO) for UK
and International astronomers. Working closely with other VO
projects worldwide through the International Virtual
Observatory Alliance (IVOA), internationally recognised
interface standards are emerging which promote true
scientifc interoperability of astronomical data and processing
resources worldwide. AstroGrid is a leading member of this
community.
![Page 22: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/22.jpg)
AstroGrid – The UK’s Virtual Observatory
Who’s Involved?
And others………
http://www.astrogrid.org/
![Page 23: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/23.jpg)
AstroGrid – The UK’s Virtual Observatory
How does it all work?
• Server side applications– Dataset Access (DSA)– Common Execution Architecture (CEA)– Community and VOSpace– Registry
• Client side application - VO Desktop– Graphical User Interface– Scripted access to the VO– Client-side workflow (Taverna)
![Page 24: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/24.jpg)
AstroGrid – The UK’s Virtual Observatory
Dataset Access - the DSA
• Provides access to catalogues
• Server application connects to Relational Database e.g. MySql, Postgres
• IVOA cone interface (positional search)
• ADQL query interface – More complex queries, similar to SQL
![Page 25: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/25.jpg)
AstroGrid – The UK’s Virtual Observatory
Common Execution Architecture - the CEA
• Provides a method for computation code to be executed– Any command-line application, although may need to
be “wrapped” in additional code
– HTTP Post/Get applications
• Limited success with executing jobs on the National Grid Service
• May look to running jobs on the UCL Condor Pool
![Page 26: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/26.jpg)
AstroGrid – The UK’s Virtual Observatory
Community and VOSpace
• Community– Used to give users an identity in VOSpace– A method of authentication for running queries/jobs
• VOSpace– Personal file storage– Attached to community identity– Can execute jobs and save results online– Can access data from any location– ivo://uk.ac.ucl.star/DuganWitherick#atable.vot
![Page 27: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/27.jpg)
AstroGrid – The UK’s Virtual Observatory
The Registry
• Perhaps most important server application• Allows astronomer or service to be able to locate,
get details of, and make use of, any resource located anywhere in the IVO space
• Two types:– Full registry - searches (or harvests) other registries
and maintains the entries in a list which is searchable by users or other services
– Publishing registry - tells full registries what is available under that authority
![Page 28: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/28.jpg)
AstroGrid – The UK’s Virtual Observatory
How everything communicates - an example
Client
Registry
DSA Server
CEA Server
VO Space
Invoke
Monitor
Notification
Read Results
Resolve
Read Inputs
Write Results
![Page 29: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/29.jpg)
AstroGrid – The UK’s Virtual Observatory
Client Side Application - The VO Desktop
• Application developed in Java• Currently in alpha form but beta
released end of month• User is presented with a single
button “Start”• When running, provides XML-RPC
interface to the Virtual Observatory
![Page 30: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/30.jpg)
AstroGrid – The UK’s Virtual Observatory
The All-VOScope
![Page 31: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/31.jpg)
AstroGrid – The UK’s Virtual Observatory
PLatform for Astronomical Tool InterConnection - PLASTIC• Communication protocol for client side virtual
observatory tool• Essentially allows tools to transmit and receive
data and instructions from other tools• Tools can instruct each other (e.g.):
– To load virtual observatory tables– Highlight subsets of rows– Load an image of a particular area of sky
• For more information visit http://plastic.sourceforge.net/
![Page 32: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/32.jpg)
AstroGrid – The UK’s Virtual Observatory
VO Explorer
![Page 33: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/33.jpg)
AstroGrid – The UK’s Virtual Observatory
VO Explorer
![Page 34: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/34.jpg)
AstroGrid – The UK’s Virtual Observatory
The Task Runner
![Page 35: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/35.jpg)
AstroGrid – The UK’s Virtual Observatory
Client-side Science Scripting
• Python - easy to learn high level programming language
import sys, xmlrpclib, os, time
prefix = file(os.path.expanduser("~/.astrogrid-desktop")).next().rstrip()
acr = xmlrpclib.Server(prefix+"xmlrpc")
ag = acr.astrogrid
userInfo = ag.community.getUserInformation()
![Page 36: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/36.jpg)
AstroGrid – The UK’s Virtual Observatory
Client Side Workflow - Taverna
![Page 37: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/37.jpg)
AstroGrid – The UK’s Virtual Observatory
Client Side Workflow - Taverna
![Page 38: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/38.jpg)
AstroGrid – The UK’s Virtual Observatory
AstroGrid is not…
• True Grid Computing– Could be loosely referred to as a data grid– Efforts are being made to get it working with true
computational grids
• A solution to the entire Virtual Observatory– Focus has been on infrastructure
• The only way for astronomers to access online data
![Page 39: AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.](https://reader036.fdocuments.in/reader036/viewer/2022062422/56649f045503460f94c17aa5/html5/thumbnails/39.jpg)
AstroGrid – The UK’s Virtual Observatory
But it….
• Is probably the most advanced of the Virtual Observatories
• Does provide an easy/heterogenous method for astronomers to access their data
• Will continue to be improved upon by the dedicated team of programmers as members of AstroGrid and other astronomers increasingly use the software and provide critical user feedback