Virtual Laboratories

VGL and FriendsR.Fraser, T.Rankine, J.Vote, R.Woodcock

AuScope Grid Roadshow 2014

CSIRO | MINERAL RESOURCES FLAGSHIP

Messy desk

2 |

– where’s your analysis?

Let’s map it out…Hardcopy of data

SSH Client

Maths PackageProprietary Package

Transform to a regular grid

Crop data to area of interest

Reformat data for processing

Upload data to HPC resource

Configure job and start

processing

Download results

Get field data Visualise data

3 |

Components of a Virtual Laboratory

Data Services

Toolboxes Compute Services

Enablers Virtual Laboratory

4 |

…to this

5 |

Virtual Geophysics Laboratory

Build “science” from existing

libraries

Run jobCollect and

publish results

Discover (raw) data

Select spatial bounds

Virtual Laboratories – Scientific Workflows

• Automated - Simplify access and integration• Reproducible process• Flexible – reusable, support differing licensing

• Allow scientists to focus on the science and not the tech• Exploiting the Cloud and HPC• Operational or education tool

6 |

Components of a Virtual Laboratory

ToolboxesData Services

Compute Services

Enablers Virtual Laboratory

7 |

Data discovery

8 |

Data discovery

9 |

Some data services support subsetting

Some data services support reformattinge.g. CSV, NetCDF, GeoTIFF

Components of a Virtual Laboratory

Data Services

Toolboxes Compute Services

Enablers Virtual Laboratory

10 |

Toolboxes & Compute Services

11 |

A variety of different Toolboxes (scientific codes) are available

Flexibility in what compute services to utilise

Toolboxes

12 |

The steps so far have been building an environment to run a processing script

...or build from existing templates

Either write your own...

Components of a Virtual Laboratory

Data Services

Toolboxes Compute Services (OGC, Provenance)

Virtual Laboratory

Enablers

13 |

Virtual Geophysics LaboratoryBuild “science” from existing

libraries

Run job

Collect and publish results

Discover (raw) data

Select spatial bounds

Published Record

Enablers/Provenance

14 |

• Standards are used to enable….”enough said”• Provenance – information that describes creation of something

• Process record• Audit trail• Publish the actual method/process used to create the results!

What have we gained?

15 |

Provenance“....documenting

their procedures .......in

VGL it is all recorded for us.”

Data Access - “Geophysicists no

longer have to spend days

searching for data from multiple

sources, downloading........ reformatting....”

Reproducibility “......collaboration and sharing now possible”

Barrier to Entry“......making geophysical

processing and modelling tools available to the

broader community at a far lower cost. It

lowers the skills barrier to entry”

Components of the Virtual Geophysics Laboratory

Data Services

Toolboxes Compute Services

Dynamic Virtual Geophysics

Laboratories

Magnetics

Gravity

DEM

eScript

Underworld

PetascaleCompute

Org.Cloud

NeCTAR Cloud

AmazonCloud

Desktop

Service Orchestration

VGLPortal

ProvenanceMetadata

ScriptingTool

eScript

Mag. Grav.Org.Cloud

VGLPortal

VGLPortal

DEM

Mag. Grav.

PetascaleCompute

Org.Cloud

Underworld

Enablers(OGC, Provenance)

16 |

Repurposing to a Virtual Hazards Laboratory (VHIRL)

Dynamic Virtual Hazards

Laboratories

Magnetics

Gravity

DEM

ANUGA

eScript

Petascale

Org.Cloud

NeCTAR Cloud

AmazonCloud

Desktop

Service Orchestration

VHIRLPortal

ProvenanceMetadata

ScriptingTool

ANUGA

Mag. Grav.Petascale

VHIRLPortal

VHIRLPortal

DEM

Bathy DEM

AmazonCloud

Org.Cloud

Landsat

Bathymetry

eScript

Unchanged

Data Services

Toolboxes Compute Services

Enablers(OGC, Provenance)

17 |

VL: Engagement

• My job (Provenance)• Reproducible process

• My toolbox (Market Place)• Analogous to cloud application store• Automation

• My Field(VL core)• Flexibility

18 |