Summary of distributed tools of potential use for JRA3

http://europlanet-ri.eu

Summary of distributed tools of potential use for JRA3

Dugan WitherickHPC Programmer for the Miracle Consortium

University College LondonJRA3 Workshop @ UCL: 22-24 April 2010

Talk Outline

2JRA3 Workshop @ UCL: 22-24/04/2010

Summarise the Goals of JRA3 Tasks 3 and 4 Task 3: Distributed Numerical Modelling Laboratory (DNML) Task 4: Distributed Data Analysis Laboratory (DDAL)

What is IDIS? What are other people doing (two examples)? A possible solution from the Virtual Observatory

Who’s using Virtual Observatory Technology The Universal Worker Service


Summary of Tasks 3 and 4 Goals (a reminder)


Task 3: The main goal of this task is to bring together the numerical simulation efforts

currently underway in Europe and to create a coordinated European multi-disciplinary modelling facility for Planetary Science: a Distributed Numerical Modelling Laboratory (DNML).

….feeding into the catalogue part of SA IDIS. Task 4:

The main goal of this task is to bring together expert teams in data analysis with their present research infrastructures and create a coordinated European interdisciplinary data analysis facility for Planetary Science: a Distributed Data Analysis Laboratory (DDAL).

….feeding into the catalogue part of SA IDIS. The main technical requirements for these tasks could be satisfied with the

same solution


Integrated and Distributed Information Service(IDIS)


What is the purpose of IDIS? The task of IDIS is to provide an easy-to-use web-based platform to give access to

available data, initiate new research activities needed to interpret observations or to solve open questions, locate teams and laboratories with special knowledge needed, and exploit synergies between space-based missions and capabilities of ground-based observatories.

A set of tools for describing, accessing and combining information and data from different sources will be developed, with the objective to offer a Virtual Observatory like access to many data essential for planetary research from European and non-European sources.

Any solution we come up with must fit in with the existing plans of IDIS


What are other people doing?(two examples)


Automated Multi Dataset Analysis (AMDA) A generic Webtool for Space Physics data

o automated event search and characterisationo catalogue generation and exploitationo automated database conditional extractiono access to remote Data Centers

Web portal located at http://cdpp-amda.cesr.fr/ The service can be reached and used from other sites according to the Virtual

Observatory paradigm. Not so much a distributed data analysis tool but a single tool linked into the

Virtual Observatory


10JRA3 Workshop @ UCL: 22-24/04/2010

Community Coordinated Modelling Centre CCMC http://ccmc.gsfc.nasa.gov/ Focus on Space Weather

Models are run through an automated request system: Instant Run – results are provided immediately or Run on Request – user is informed by email when run has completed

o These longer running models are added to a queue to run on their Beowulf Cluster

Capable of running more than one model/analysis tool Not really distributed

Appears that all work is being completed at one site

11JRA3 Workshop @ UCL: 22-24/04/2010

12JRA3 Workshop @ UCL: 22-24/04/2010

13JRA3 Workshop @ UCL: 22-24/04/2010

Remember IDIS (Distributed is a keyword)

14JRA3 Workshop @ UCL: 22-24/04/2010

15JRA3 Workshop @ UCL: 22-24/04/2010

A Possible Solution from the Virtual Observatory

16JRA3 Workshop @ UCL: 22-24/04/2010

Common Execution Controller (CEC) Web service for running various asynchronous applications. Part of the Common Execution Architecture (CEA) suite of software from Astrogrid. Designed to wrap existing applications so that they are presented as IVOA standard

web services.o Commandline applicationso Existing web form based HTTP applicationso Custom written Java applications

Suitable for applications that:o have no graphical interfaceo need no user interaction once launch

Supports the following standards: UWS, VOSI, General Security, Registry, Delegation, VOSpace

JRA3 Workshop @ UCL: 22-24/04/2010 17

The Universal Worker Service (UWS) An International Virtual Observatory Specification http://www.ivoa.net/Documents/UWS/ The Universal Worker Service (UWS) pattern defines how to manage asynchronous

execution of jobs on a service Has features over the previous standards of

o Control for job lifetimeo Access to user sandbox (in order to inspect intermediate files

There are UWS deployments that do not use the CEAo Not necessary to use CEA

18JRA3 Workshop @ UCL: 22-24/04/2010

Virtual Observatory Registry Essentially the “Yellow Pages” or Telephone Directory for services User client (web portal or client tool) is used to search the registry

o Information about the resource is returned Brief summary Contact Name Inputs for the model/tool Location for the model/tool

Two types of registryo Local registry which lists the services installed locallyo Full registry harvests the local registries to find out what they have installedo Users search the full registry

JRA3 Workshop @ UCL: 22-24/04/2010 19

JRA3 Workshop @ UCL: 22-24/04/2010 20

JRA3 Workshop @ UCL: 22-24/04/2010 21

Advantages to this plan Allows us to take advantage of the extensive effort made in developing standards

o Saves us a lot of effort We can use/modify existing VO compliant tools

o Tool for OPerations on Catalogues And Tableso AladinVO – VO compliant image analysis toolo VOSpec – spectral analysis tool

Possible access to workflow tools Provides us with VO access to other data/tools Some IDIS data sources are already published to the VO

Disadvantages to this plan Unclear whether this would be supported by IDIS The VO has had its own problems which we may end up getting caught in This may not be appropriate for fulfilling our deliverables

JRA3 Workshop @ UCL: 22-24/04/2010 22

Summary of distributed tools of potential use for JRA3

Documents

Transcript of Summary of distributed tools of potential use for JRA3