eSDO: UK Access to Data and Algorithms from the Solar ...

eSDO Algorithms, Visualization and Data Access

Elizabeth Auden27 September 2006HELAS WorkshopNice

27 September 2006 HELAS Roadmap Workshop Elizabeth Auden

What is eSDO?eSDO is an e-science project funded by PPARC to make SDO data

and algorithms available to the UK solar community using the virtual observatory.

eSDO Duration: 36 months, October 2004 – October 2007SDO Launch: October 2008

RESEARCH - Phase A: 1 October 2004 – 30 September 2005IMPLEMENTATION - Phase B: 1 October 2005 – 30 September 2007

Current eSDO Project Documentation:http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO

http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO

http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO


eSDO Workpackages

I. Algorithms• Code: feature recognition, local helioseismology, global

helioseismology• Grid integration: AstroGrid and JSOC

II. Data centres• AstroGrid interface with US JSOC data centre• AstroGrid interface with UK (or European?) data centre

III. Visualization tools• Catalogues• SDO Streaming tool• On-the-fly image galleries and movies


eSDO Project Team• MSSL: Coronal Image Analysis

– Elizabeth Auden, project manager– Len Culhane, PI– Lidia van Driel-Gesztelyi– Mike Smith

• RAL: Event and Feature Recognition– Richard Harrison– Andrzej Fludra– Vito Graffagnino

• Sheffield: Local Helioseismology– Mike Thompson– Sergei Zharkov

• Birmingham: Global Helioseismology– Yvonne Elsworth– Bill Chaplin– Thierry Toutain

US Links:

JSOC, Joint Operations Centre

Stanford University

Lockheed Martin


AlgorithmsCodeFeature Recognition / Event Analysis

• Coronal loop detection, MSSL• Non-linear magnetic field extrapolation, MSSL• Helicity computation, MSSL• Small event detection, RAL• CME dimming region recognition, RAL

Local Helioseismology• Perturbation map generation, Sheffield• Subsurface flow analysis, Sheffield• Local helioseismology inversions, Sheffield

Global Helioseismology• Mode parameters analysis, Birmingham• Mode asymmetry analysis, Birmingham

Solar image from TRACE mission. Coronal loops marked in blue.


AlgorithmsGrid Integration

JSOC• SDO pipeline at Stanford University / Lockheed Martin• eSDO algorithms deployed as pipeline modules (automated or user-run)• Data will be obtained locally from JSOC archive

AstroGrid CEA• UK hosted algorithms hosted as web services• Accessible through AstroGrid workflows• Data transferred from MySpace, remote data centre or user’s local storage

SolarSoft• Algorithm C modules wrapped in IDL• Distribution through MSSL SolarSoft gateway• Data transferred to algorithm from user’s local storage


AlgorithmsAstroGrid Deployment

9 algorithms have been deployed as CEA applications


Data CentresUS Data CentrePrimary US Data Store• Files stored in SUMS (Storage Unit Management System) at Stanford

University with metadata in distributed DRMS (Data Resource Management System)

• DSA / DRMS integration for grid data access investigations• VSO (Virtual Solar Observatory) primary US access

AstroGrid / VSO Interface• VSO queries in AstroGrid workflows through STAP (Simple Time Access

Protocol) web service

US / UK Network Latency Tests• Test expediency of file transfer mechanisms• Test between Stanford, MSSL, UCL, RAL


Data CentresPossible UK Data Centre

Storage • 30 TB disc cache• 60 day rolling cache HMI/AIA• True cache for HMI / AIA• DRMS / SUMS

Transfer • Export formats from JSOC• Data search – DSA• Data transfer - CEA • Test installation MSSL

Investigation of DRMS / SUMS


Data CentresPossible European Data Centre

• EU Framework 6 support for European helioseismology data and algorithms

• Possible full mirror of SDO data in Europe?• DSA / DRMS interface for AstroGrid access

to full mirror of SDO data in Europe?

DRMS DRMSDRMS

SUMS SUMSDSA UK datasetsUS UK Europe

TLAs:DRMS: Data Resource Management System (JSOC)

SUMS: Storage Unit Management System (JSOC)

DSA: DataSet Access (AstroGrid)


Note: AstroGrid components (JES, DSA, MySpace, Workbench) may be located anywhere.

Data Centres eSDO / Solar-B Prototype: system perspective

UKSSDCADS file system

(Local)

AstroGrid

HelioScope

(Local)

URLsSTAP serviceXML


Data Centres User access through AstroGrid

HelioScope: solar data results from Simple Time Access Protocol services.

Task Launcher: locate DataSet Access components, submit ADQL queries

STAP service will point to URLs in UK, US or Germany.

DSA will integrated with DRMS (Data Resource Management System) in UK or Germany.

AstroGrid workbench: STAP service access through HelioScope, DSA access through the Task Launcher.


Visualization ToolsCatalogues

Small Event / CME Dimming catalogue• Generated from continuously running of small event detection / CME dimming

region recognition algorithms• Tables of event / CME statistical information• Likely to be generated in UK

Helioseismology Mode Parameters catalogue• GONG month of data generated from mode parameters analysis algorithm• Likely to be generated in US

Catalogues can be accessed through ADQL (Astronomical Data Query Language) query to DSA.


Visualization ToolsSDO streaming toolUser Facilities• Pan in space • Zoom in space • Pan in time• Zoom in time

Data Products• LoS Magnetogram• HMI continuum map• AIA images

Future Work• FFT / wavelet analysis for

coronal wave investigations


Visualization ToolsOn-the-fly Image Galleries and MoviesThumbnail Extraction • Extract thumbnails in US

from HMI and AIA science products

Image Gallery Generation • User specifies start time,

stop time, cadence, data product

• Image gallery generated on the fly in web browser

Movie Generation• User specifies start time,

stop time, cadence, data product

• MPEG movie generated on the fly

Find data with AstroGrid HelioScope.

Use PLASTIC to load data into visualization tools.


US-funded Virtual Observatory Infrastructure to search, obtain and analyze solar data

eSDO and the Virtual Observatory

eSDO has links with AstroGrid, VOTech, VSO, and the IVOA.

UK-funded e-science project Provide access to SDO data, algorithms and visualization tools through AstroGrid.

VOTech: EU Framework 6 funded Technology / infrastructure for European virtual observatory

UK-funded Virtual Observatory Infrastructure for the UK: data access, applications, virtual storage, workflows, computational resources

Int’l Virtual Observatory Alliance Defines standards for VO schemas and coordinates interoperable software between virtual observatories


Future WorkPhase B implementation development now in progress:• Summer / Autumn 2006:

– Algorithm integration with AstroGrid; test integration with JSOC– Unit and science testing for algorithms– Add STAP service client to visualization tools– Add movie functionality to streaming tool– Investigate DRMS integration with DSA

• 2007: – Completed algorithms– Full algorithm integration with JSOC and SolarSoft– Coronal wave analysis in SDO streaming tool– Work with HELAS to explore European SDO data centre– Catalogues


Further Information

LinkseSDO: http://www.mssl.ucl.ac.uk/twiki/bin/view/SDOAstrogrid: http://www.astrogrid.orgVOTech: http://wiki.eurovotech.org/bin/view/VOTech/IVOA: http://www.ivoa.net

Software to tryAstroGrid Workbench: http://www2.astrogrid.org/desktop/indexHelioScope: included with workbencheSDO Streaming Tool:

http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/StreamingTool

ContactseSDO: Elizabeth Auden, [email protected] Solar Science: Silvia Dalla, [email protected]

http://www.astrogrid.org/

http://wiki.eurovotech.org/bin/view/VOTech/

http://www.ivoa.net/

mailto:[email protected]

eSDO: UK Access to Data and Algorithms from the Solar ...

Technology

Transcript of eSDO: UK Access to Data and Algorithms from the Solar ...