Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and...

22
Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure (IMDI) and the World Data Center for Climate (WDCC) Michael Lautenschlager + Hannes Thiemann Modelle und Daten Max-Planck-Institut für Meteorologie, Hamburg Hamburg, 29.03.06

Transcript of Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and...

Page 1: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1

Introduction Course 2006Services and Facilities of DKRZ and M&D

Integrating Model and Data Infrastructure (IMDI)and the

World Data Center for Climate (WDCC)

Michael Lautenschlager + Hannes ThiemannModelle und Daten

Max-Planck-Institut für Meteorologie, Hamburg

Hamburg, 29.03.06

Page 2: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 2

• WDCC

• IMDI

Page 3: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 3

IMDI + WDCC

Page 4: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 4

Central Components of IMDI

SCE: Standard Compile Environment for flexible model assemblage and compilation

SRE: Standard Run Environment for efficient use of HPC architectures

WDCC: World Data Center for Climate for long-term archiving and international, interdisciplinary data dissemination according to the rules of the ICSU WDC system

Page 5: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 5

Objective and requirements

IMDI objective: Integrate any earth system model on any platform

…. requires standards for .... Source Codes Component models Coupling software Compilation environment Running environment Quality assessment Postprocessing environment Visualization environment

Page 6: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 6

Benefits from IMDI

Common look-and-feel for all models Common look-and-feel for all platforms Portability profit from future developments Support (M&D, PRISM Team, vendors) Wide community of users Facilitation of model exchange

within a coupled configuration

between institutes

Post-processing, visualisation tools

Page 7: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 7

OCEAN

Complexity of Earth System Models in IMDI

ATMOSPHERE

Atmospheric Chemistry

Sea Ice

OASIS3

MarineBiogeo-

chemistry

LandSurface

Page 8: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 8

Model configurations adapted to the SCE and SRE

(Coupled) Model Component modelsECHAM5 ECHAM5

ECHO (=MPI-AO) ECHAM5 + MPI-OM

EOL ECHAM5 + OPA+LIM

MPI-AOB ECHAM5 + MPI-OM+HAMOCC

MPI-AO-PISCES ECHAM5 + MPI-OM+PISCES

MPI-OM MPI-OM

MPI-OB MPI-OM+HAMOCC

MPI-O-PISCES MPI-OM+PISCES

IPSL_CM4 OPA+LIM + LMDZ+ORCHIDEE

OPATOY OPA+LIM + TOY4OPA

TOYCLIM TOYATM + TOYOCE + TOYCHE

Page 9: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 9

PRISM Platforms

Linux i686 divers inst.

Cray X1 Cray, France

IBM Power4 ECMWF, Reading

Fujitsu Opteron Cluster F.S.E. Toulouse

Fujitsu VPP5000 Meteo France, Toulouse

NEC SX-5 IPSL, Paris

NEC SX-6 e.g. DKRZ, Hamburg

SGI Altix 3000 e.g. CERFACS, Toulouse

SGI IRIX64 SGI, Munich

SGI Origin 3800 KNMI, De Bilt

Page 10: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 10

SCE: Model Integration

Page 11: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 11

Experiment Flow in SCE + SRE

CVS Repository

Retrieving the code

Source code modifications

Compile script generation Compilation

Experiment configuratio

n

Model Execution

Output Data Post-processing Visualisation Archiving

Tasks generation

Page 12: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 12

Standard Running Environment

Create_TASKS.frm cplmod expid [node]

model executionRUN_cplmod_expid

postprocessing

visualization

archiving

POST_cplmod_expid

VIS_cplmod_expid

SAVE_cplmod_expid

executablesmod1_ID.MPI[1,2].frmmod2_ID.MPI[1,2].frm

oasis3.MPI[1,2].frm

Input data

archiveoutput dataRestart files

Page 13: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 13

SRE: Directory Structure

m od 1 _ id .M P I2 .xm od 2 _ id .M P I2 .x...oas is3 .M P I2 .x

bin

m od 1 log...* .p rt*cp lou t

log

fileAfileBfileC

m od1

fileAfileB

m od2

...

n am cou p leg rid s .n c...

oasis3

input

res t1 _ e n d d a te

m od1

res t2 a_ e n d d a teres t2 b _ e n d d a te

m od2

...

ss tocean...

oasis3

restart

fileAfileB

m od1

fileAfileBfileC

m od2

...

VAR _ ou t.d a te. . .

oasis3

outdata

expid

PR ISMR O O T

Page 14: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 14

The Runscript

comments_cplmod.hqueue_commands_node.hsetup_cplmod_expidcomplete_setup_cplmod.hfunctions.hcalendar.hcreate_directories.hget_executables_cplmod.hget_input_data_model.hnamelist_model.hlaunching_cplmod_node.hsave_output_model.hsubmit_next_jobs.h

for all component models and the

coupler

Definitions

Preprocessing

ExecutionPostprocessingJob submission

cplmod: coupled modelnode: node nameexpid: experiment IDmodel: component model

Page 15: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 15

Start: Approved in January 2003

Maintenance: Model and Data (M&D/MPIMET) and German Climate Computing Centre (DKRZ)

Mission: Data for climate research are collected, stored and disseminated

ICSU Policy: long-term archiving and unrestricted data access for scientists

Restriction: Only climate data products in CERA DB, no raw data storage.

Content: Emphasis is on climate modelling and related data products.

Co-operation: with thematically corresponding data centres like WDC-MARE (Bremen) and WDC-RSAT (Oberpfaffenhofen)

URL: http://www.wdc-climate.de/

WDCC

The WDCC is IMDI's central component for data archiving and data dissemination

Page 16: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 16

WDCC Content

Data from Earth SystemModelling andRelated Observations

March 2006: 220 TB / 566 Experiments / 77.000 Data Sets

ERA40

IPCC

CEOP

BALTEX

HOAPS

CARIBIC

WOCE

ERA15/40NCEP

GEBCO

COSMOS

Simulations @ MPI, GKSS,…

EH5/MPI-OMIPCC-AR4

ENSEMBLES

IPCC-DDC

Page 17: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 17

WDCC Data Catalogue… based on the CERA-2 data model

Entry

Reference

Status

Distribution

Contact Coverage

Parameter

SpatialReferenceLocal Adm.

Data Access

Data Org

Page 18: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 18

WDCC Database Size

March

Database Size

0

50

100

150

200

250

1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

TB

yte

Size

March

Page 19: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 19

WDCC Usage

Downloads per Year

0

100000

200000

300000

400000

500000

600000

700000

1999 2000 2001 2002 2003 2004 2005

Page 20: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 20

WDCC user distribution

Page 21: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 21

Page 22: Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 1 Introduction Course 2006 Services and Facilities of DKRZ and M&D Integrating Model and Data Infrastructure.

Lautenschlager + Thiemann (M&D/MPI-M) / 29.03.06 / 22

Further information

M&D's webpage:

http://www.mad.zmaw.de

WDC Climate:

http://www.wdc-climate.de

Assistance:

[email protected]

[email protected]


Z EGU2015-8404 Integration of external metadata into the Earth System Grid Federation (ESGF) K. Berger 1, G. Levavasseur 2, M. Stockhause 1, and M. Lautenschlager.

Z EGU2015-8404 Integration of external metadata into the Earth System Grid Federation (ESGF) K. Berger 1, G. Levavasseur 2, M. Stockhause 1, and M. Lautenschlager.

QA-DKRZ: The Annotation Model · QA-DKRZ Tool • Work-flow • Dependencies Annotation Model • Specification of actions tagged to checks • Structure of results: Files and directories

QA-DKRZ: The Annotation Model · QA-DKRZ Tool • Work-flow • Dependencies Annotation Model • Specification of actions tagged to checks • Structure of results: Files and directories

PID usage at DKRZ, the role of RDA and ePIC policies

PID usage at DKRZ, the role of RDA and ePIC policies

High-Performance I/O @ Mistral fileHigh-Performance I/O @ Mistral Julian M. Kunkel kunkel@dkrz.de German Climate Computing Center (DKRZ) 06-10-2015

High-Performance I/O @ Mistral fileHigh-Performance I/O @ Mistral Julian M. Kunkel [email protected] German Climate Computing Center (DKRZ) 06-10-2015

M. Lautenschlager (M&D/MPIM) / 24.04.2003 / 1 WDC Konsortium Meeting Oberpfaffenhofen, 24/25.04.03 1)CERA-DB 2)WDCC Homepage 3)DLR EOWEB Zugang 4)Kostenmodell.

M. Lautenschlager (M&D/MPIM) / 24.04.2003 / 1 WDC Konsortium Meeting Oberpfaffenhofen, 24/25.04.03 1)CERA-DB 2)WDCC Homepage 3)DLR EOWEB Zugang 4)Kostenmodell.

Jan Brase Learning Lab Lower Saxony, Uni. Hannover Michael Lautenschlager WDC for Climate

Jan Brase Learning Lab Lower Saxony, Uni. Hannover Michael Lautenschlager WDC for Climate

Data Standardization at DKRZ - ECMWF...Data Standardization at DKRZ • Overview 1: DKRZ data management! • ESGF conditions for Data Admission! • What has to be specified?! •

Data Standardization at DKRZ - ECMWF...Data Standardization at DKRZ • Overview 1: DKRZ data management! • ESGF conditions for Data Admission! • What has to be specified?! •

9-Sept-2003CAS2003, Annecy, France, WFS1 Distributed Data Management at DKRZ Distributed Data Management at DKRZ Wolfgang Sell Hartmut Fichtel Deutsches.

9-Sept-2003CAS2003, Annecy, France, WFS1 Distributed Data Management at DKRZ Distributed Data Management at DKRZ Wolfgang Sell Hartmut Fichtel Deutsches.

Data near processing support for - DKRZ · Data near processing support for climate data analysis . Stephan Kindermann (DKRZ) EGU 2016 Overview Background / Motivation Climate community

Data near processing support for - DKRZ · Data near processing support for climate data analysis . Stephan Kindermann (DKRZ) EGU 2016 Overview Background / Motivation Climate community

CMIP6 Data Management at DKRZ - CISL Home · CMIP6 Data Management at DKRZ . iCAS2015 . Annecy, France on 13−17 September 2015. Michael Lautenschlager . Deutsches Klimarechenzentrum

CMIP6 Data Management at DKRZ - CISL Home · CMIP6 Data Management at DKRZ . iCAS2015 . Annecy, France on 13−17 September 2015. Michael Lautenschlager . Deutsches Klimarechenzentrum

DKRZ System and European Commission Center of Excellance in ...

DKRZ System and European Commission Center of Excellance in ...

Big Data Services at DKRZ - Max Planck Society€¦ · Big Data Services at DKRZ Michael Lautenschlager and Colleagues from DKRZ and Scientific Computing Research Group MPI-M Seminar

Big Data Services at DKRZ - Max Planck Society€¦ · Big Data Services at DKRZ Michael Lautenschlager and Colleagues from DKRZ and Scientific Computing Research Group MPI-M Seminar

M. Diepenbroek (MARUM), M. Lautenschlager (MPI-M), E. Paliouras (DLR), H. Grobe (AWI) CODATA General Assembly, Berlin 10.11.2004 World Data Center Cluster.

M. Diepenbroek (MARUM), M. Lautenschlager (MPI-M), E. Paliouras (DLR), H. Grobe (AWI) CODATA General Assembly, Berlin 10.11.2004 World Data Center Cluster.

Cunningham, J. A. , Rahman, I. A., Lautenschlager, S ... · A virtual world of paleontology John A. Cunningham, Imran A. Rahman, Stephan Lautenschlager, Emily J. Rayfield, and Philip

Cunningham, J. A. , Rahman, I. A., Lautenschlager, S ... · A virtual world of paleontology John A. Cunningham, Imran A. Rahman, Stephan Lautenschlager, Emily J. Rayfield, and Philip

Workshop M&D / DKRZ 22.-24. 02. 2005 Jörg Wegner Dataformats, Postprocessing and Visualization Content: 1)Dataformats: Grib, NetCDF 2)Tools for Postprocessing:

Workshop M&D / DKRZ 22.-24. 02. 2005 Jörg Wegner Dataformats, Postprocessing and Visualization Content: 1)Dataformats: Grib, NetCDF 2)Tools for Postprocessing:

Long-term Archiving of Climate Model Data at WDC Climate and DKRZ Michael Lautenschlager WDC Climate / Max-Planck-Institute for Meteorology, Hamburg Data.

Long-term Archiving of Climate Model Data at WDC Climate and DKRZ Michael Lautenschlager WDC Climate / Max-Planck-Institute for Meteorology, Hamburg Data.

DKRZ German Climate Computing Center

DKRZ German Climate Computing Center

DKRZ and M&D Introduction course 2006 Diagnostic & Visualisation Introduction Diagnostic cdo Visualisation gnuplot GrADS GMT cdat.

DKRZ and M&D Introduction course 2006 Diagnostic & Visualisation Introduction Diagnostic cdo Visualisation gnuplot GrADS GMT cdat.

Compressed Instruction Cache Prepared By: Nicholas Meloche, David Lautenschlager, and Prashanth Janardanan Team Lugnuts.

Compressed Instruction Cache Prepared By: Nicholas Meloche, David Lautenschlager, and Prashanth Janardanan Team Lugnuts.

M.Lautenschlager (WDCC / MPI-M) / 08.12.05 / 1 AGU Fall Meeting, San Francisco, December 2005 Michael Lautenschlager - WDC Climate (Max-Planck-Institut.

M.Lautenschlager (WDCC / MPI-M) / 08.12.05 / 1 AGU Fall Meeting, San Francisco, December 2005 Michael Lautenschlager - WDC Climate (Max-Planck-Institut.