Towards Automated Model Output Analysis

Charles Doutriaux

Ispra 2006

Summary

• The Problem

• PCMDI’s Experience– Overview– CDAT– ESG– Collaborations

• The AutoMOD Project

• Portability

The Climate Model Data Management Issue

• Data– Different formats – netCDF is not standardized

• Different sites require knowledge of different methods of access

• Metadata– Painful to produce– Most kept in files separate from data– unsearchable unless one is “in the

know” (some kept in people’s brains)• Access control

– Manual– Not formalized

• Data requests/analysis– Beginnings of a formal process

Beginnings of web portals– Far too much done by hand– Logging nearly non-existent

• Data– netCDF is standardized to the CF model– Different sites but standardized access

protocol.

• Metadata– Created via batch or semi-automated

processes– Kept in databases and readily

searchable • Access control

– Formalized– Highly granularized – down to per-file,

per-person level• Data requests/analysis

– Completely automated– All logging done automatically

Usually Today, Tomorrow

Computers do nearly all the work and scientists do more interesting things than shoving bytes around

Tremendous manual intervention, horribly inefficient by any measure, resource wasting

Proposed SolutionAutoMOD

Automated Model Diagnostic Facility • Web based-Interface• Automated Upload• Automated Atlases (pre-run/offline diagnosis)• Online Diagnosis• Searchable Database for Model, Simulations,

Diagnosis• Leverage from PCMDI’s experience

PCMDI

• Goal– Serve Climate Community

• Computation Team Goals– Provide the scientific community with tools to

allow them to focus on science NOT on technical aspects.

PCMDI Solutions

• Analysis: Climate Data Analysis Tools

• Data serving: ESG

Climate Data Analysis Tools Design/Philosophy

• Leveraged from community’s work, for the community• Cutting-EdgeTechnology

– Tomorrow’s Analysis Tools Today.

• Quality driven but balanced by functional requirements. • Python based

– Flexible– Efficient

• Open Source, Open Community.– Knowledge sharing– Time saving– Leveraging from others work

Climate Data Analysis Tools (CDAT)

CDAT

Climate Data Analysis Tools• Python based system• Added packages by community• One environment, • Community Software

CDAT’s Modularity

Scripts / VCDAT

XM

GR

AC

E

VC

S

VT

K

Python

NE

TC

DF

PP

HD

F4

GrA

DS

CD

MS

DR

S

XM

L

Cdunif.soGraphics method

BO

XF

ILL

ME

SH

FIL

L

ISO

LIN

E

ISO

FIL

L

OU

TF

ILL

OU

TL

INE

YX

vsX

XY

vsY

XvsY

TA

YL

OR

SC

AT

TE

R

VE

CT

OR

_vcs.so

CanvasC o

r Fo

rtran

CommunityContributed Packages

f2pyPyfort

CDMS

Dataset

Numeric

MVMA

VariableAxisGrid

CDATExamples

Earth System Grid

• Collaboration and data sharing• Location-independent equal-access to shared

resources – (data, visualization, supercomputers, experiments,

whiteboard, etc..)• Evolution from centralized data sharing to

distributed data-sharing.• Allow for geographically distributed teams.• Allow researchers to focus on science not data

set manipulation.

Federated MetadataCatalog

User InterfaceServer

ESG Virtual Server

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Federated ESG Sites

ESG Product Request Protocol Publish to Federated Catalog

HTTP

Catalog

Data

ESG Node

Site 2

Catalog

Data

ESG Node

Site 1

Catalog

Data

ESG Node

Site 3

Catalog

Data

ESG Node

Site 4

External collaborationsPortal examples

GeoSPLATBritish Atmospheric Data Centre

Live Access ServerPacific Marine Environmental Laboratory

AutoMOD Automated Model Diagnostic

Facility • Freely available

• Automated Upload

• Automated Atlases

• Searchable Database for Model, Simulations, Diagnosis

• System can evolve to incorporate new/future standard

Existing StructureThe “manual” approach

Model Output Archive

Analysis scripts (python/CDAT,

IDL, Ferret, Tecplot)

Analysis/MIP Guru(s)

User

AutoMOD StructureThe “Automated” approach

Model Output Archive

Web Server(powered with CDAT)

User

Simple HTTP

Database

FTP

AutoMODMySQL Database

• Variable information• Modeling Groups Information

– Various versions information

• Simulations Information (associated with a model version)

• Working Groups Information• Unix-like Read/Write Authorization, per

Group/User, Model version or simulation level

AutoMOD Web Interface

• Apache Server, with builtin CDAT via mod_python module

• MySQL interface– Permissions– Archive Content– AutoMOD Project Info (users, groups, etc…)

• Online CDAT Diagnosis– Allows to restrict ressources available to the user

(server swamping issues)– Use pre-loaded CDAT/Python (faster)– System 100% free and Open-Source

AutoMOD Web Interface

AutoMOD Web Interface

AutoMOD Web Interface

AutoMOD Web Interface

AutoMODUsers Requirements/Restrictions

• Data/Metadata must adhere to strict standards– NetCDF format, CF compliant– http://www.cgd.ucar.edu/cms/eaton/cf-metada

ta

• Data must be pre-processed by user via “output” subroutine, provided to them.

• No “user-defined” Diagnosis

AutoMODStatus and Upcoming Tasks

• Web administrative interface finished, only needs “look and feel” finish.

• OCMIP2 Data in the system

• First basic diagnosis in beta phase

• Extensive “Atlas” and “online” diagnosis will be added through the year.

• Possibility to move to ESG Data serving

Portability

• CF Compliant data should plug-in without any changes.

• Potential changes could be:– Project specific assimilation script– Adding/Removing Variable information stored

into the database

• Adding project specific diagnosis will obviously be needed

PortabilityPossible Extensions

• Replace Data archive system, with Earth System Grid.

• Not needed by AutoMOD at the moment, but would be for projects such as IPCC (huge datasets, more stringent security requirements)

Conclusions

• AutoMOD: Hassle-free environment– Leveraged from PCMDI’s experience:

• Powerful CDAT as diagnosis backend• Possibility of use of ESG system Easy upload of

dataset

– Centralized resources– Easy User Interface– Ever growing list of diagnosis– Easily Portable