Chad BerkleyNCEAS

National Center for Ecological Analysis and Synthesis (NCEAS),University of California Santa Barbara

Long Term Ecological Research Network Office, University of New MexicoUniversity of Kansas

San Diego Supercomputer Center

Kepler: A Workflow Tool for Heterogeneous Ecological Data

Analysis

http://seek.ecoinformatics.org December 4, 2003Edinburgh, Scotland

Outline

Quick history SEEK overview Ecological Metadata Language Using workflows in Ecology Workflow editing with Kepler Future visions

History

Late 1990s – patterns noticed in the problems surrounding data synthesis at NCEAS

1999 - Michener et al paper on ecological metadata

2000 – Knowledge Network for Biocomplexity Morpho, Metacat, Ecological Metadata Language Some footholds into workflow creation and execution

2003 – Scientific Environment for Ecological Knowledge (SEEK) Grant Continues the work done on the KNB grant Emphasis on using metadata for advanced data

processing

SEEK approach

General approach to specific ecological problems

Data described with adequate metadata in a grid accessible repository

Reasoning engine (ontology based) to locate and extract data and processes

Modeling system to put it all together and control execution flow

SEEK Components

Ecogrid Analysis Library Metadata and data repository

Semantic Mediation System Controlled semantic vocabulary Ontological discovery system

Analysis and Modeling System (Kepler) Workflow control system Utilizes resources from other components

SEEK Architecture

Ecological Metadata Language

Common language for archiving and transport of datasets

XML based Designed for/by the ecological

community Describes physical and logical

structure of data Also includes project, literature and

software information SEEK will add semantic information

Workflows in SEEK

In the SEEK model, data ingestion/cleaning is metadata driven (specifically with EML)

Output generation includes creating appropriate metadata

The analysis pipeline itself becomes metadata

Metadata driven data ingestion

Key information needed to read and machine process a data file is in the metadata File descriptors (CSV, Excel, RDBMS, etc.) Entity (table) and Attribute (column)

descriptions Name Type (integer, float, string, etc.) Codes (missing values, nulls, etc.) In the future, this will include semantic typing

Metadata revision

Metadata is revised following any transformation

Versioning of metadata and data is very important

This process results in a lineage of the data file as it has been transformed

Typical ecological workflow example

Workflows can automate the integration process if data is described with adequate structured metadata

Homogeneous data integration

Integration of homogeneous or mostly homogeneous data via EML metadata is relatively straightforward

Heterogeneous Data integration

Integration of heterogeneous data requires much more advanced metadata and processing

Attributes must be semantically typed Collection protocols must be known Units and measurement scale must be known Measurement mechanics must be known (i.e. that

Density=Count/Area)

Label data with semantic types Label inputs and outputs of analytical components with semantic types

Use Semantic Mediation System (SMS) to generate transformation steps Beware analytical constraints

Use SMS to discover relevant components Ontology – specification of a conceptualization (a knowledge map)

Semantic typing and ontologies

Data Ontology Workflow Components

Measurement Ontology

Density is part of a larger measurement ontology SEEK’s intent is to create one or more community created

ecological ontologies Creates a controlled vocabulary for ecological metadata More about this in Bertram’s talk

About Kepler

Kepler is the name of the SEEK/SDM additions to the Ptolemy modeling system

Ptolemy was designed by the UC Berkeley EECS department

Primary use is modeling EE circuits Free, opensource, pure Java Flexible design GUI for building

workflows

Kepler

A Kepler model consists of linked “actors” (which correspond to workflow steps)

Timing is controlled by a “director” All actors are written in Java but can

call other applications (such as SAS and MATLAB or native language code via JNI)

Actors can call arbitrary Web (or Grid) Services

Ptolemy already has a very large inventory of actors

Easy to use, drag ‘n drop interface

SEEK Contributions to Kepler (so far)

EML data ingestion actor

Actor design tool

EML data ingestion actor

Ingests any data format described by EML metadata

Converts raw data to Kepler format Data can then be operated on with other

actors Produces one output port for each attribute

in the dataset Individual attributes can then be mapped to

other actors

Ptolemy model with EML ingestion actor

SEEK Contributions to Kepler (so far)

EML data ingestion actor

Actor design tool

Actor design tool

Allows “place-holder” actors to be defined on the fly by non-programmers during workflow creation

Domain scientists can thereby create workflows without programming knowledge

Workflows created with these actors can be executed once their functionality is implemented by a programmer

Allows quick prototyping of workflows by domain scientists

“Place-holder” actors can still be linked to other working actors

Ptolemy and dynamically created actor

How domain scientists will benefit

More fully automated integration systems

A library of pre-defined analytical processes which can be executed on heterogeneous data

Semantic data discovery and processing

Automated unit and measurement scale conversions

A fuller understanding of cross site research implications

Acknowledgements

This material is based upon work supported by:

The National Science Foundation under Grant Numbers 9980154, 9904777, and 0225676 to NCEAS and its collaborators.

The National Center for Ecological Analysis and Synthesis, a Center funded by NSF (Grant Number 0072909), the University of California, and the UC Santa Barbara campus.

Primary Collaborators: University of New Mexico (Long Term Ecological Research Network Office), San Diego Supercomputer Center, University of Kansas (Center for Biodiversity Research)

More info: http://seek.ecoinformatics.org

Questions? IRC: irc.ecoinformatics.org #seek