The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services...
-
Upload
oliver-bruce -
Category
Documents
-
view
217 -
download
0
Transcript of The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services...
![Page 1: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/1.jpg)
The CERA2 Data Base
Data input – Data output
Hans Luthardt
Model & Data/MPI-M, Hamburg
Services and Facilities of DKRZ and Model & DataHamburg, 29.3.2006
![Page 2: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/2.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 2
Data Input
![Page 3: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/3.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 3
Including data into CERA
Data sets from users can be included into the data base and madeaccessible (by internet)
Inclusion is done by Model & Data if following conditions are met:
• data are of general interest• data should become available to a broader community• data size is small relative to the data archive size • a complete set of meta data (fitting the CERA standard) must be
provided
![Page 4: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/4.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 4
Store datadata: data must be provided in a structure that can be
transferred into the data base without further processing
meta data: XML-files for the experiment and each of the data setsmust be provided
XML-templates are available from :
http://cera-www.dkrz.de/Meta_Fill/ orhttp://cera-www.dkrz.de/Meta_Fill/mod_metadata.html
![Page 5: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/5.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 5
Preferred storage structure for 3-d gridded data:
• single variable• single level • time series• grib formatted / netCDF formatted (/ others)
Data structures
Application related data structure (2-D)(180 KByte/blob)
original data structure (4-D)(3 MByte/blob)
![Page 6: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/6.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 6
Retrieve data sets
![Page 7: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/7.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 7
Retrieve data sets
Database content:
220 TByte data volume
650 experiments80 000 data sets
Database content:
220 TByte data volume
650 experiments80 000 data sets
![Page 8: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/8.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 8
Data and Meta dataCERA Metadata Model
![Page 9: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/9.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 9
Search strategies
Keyword Project
Experiment
Dataset
Information
Download
Variable
specific Dataset(by name)
Postprocessing
Code
![Page 10: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/10.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 10
Access to the data base
Access to the CERA data base:
• by java servlet - using an internet browser• by java applet (java runtime required) - obselet!• by java command line tool (jblob )
Unrestricted browsing for meta data
User account required for data download
Start: http://www.mad.zmaw.de/
![Page 11: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/11.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 11
Access to the data base
Retrieval by script :
• Java based – for Unix/Linux and Windows
• http and JDBC connection
examples :jblob –datasetname <name> -optionsjblob –datasetid <id> -optionsjblob –showdatasets ”<search_string>” (% as wildcard)
examples :jblob –datasetname <name> -optionsjblob –datasetid <id> -optionsjblob –showdatasets ”<search_string>” (% as wildcard)
![Page 12: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/12.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 13
Access to the data base
Access by java applet:(obsolet!)
e.g.hierarchical topic search
![Page 13: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/13.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 14
Browse/Retrieve Metadata
see wini.wdc-climate.de
Metadata access via WWW:
• xsql query to DB
• xml output from DB
• xsl mapping to any metadata format
raw xml
xhtmlISO xml
DC xml
... variousmetadataformats
http:XML
xsl –mappi
ngxs
ql –
quer
ylnternetApplicati
onServer
request: URL
user applications
![Page 14: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/14.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 16
Search for data
The servlet interface currently allows search for:
• Project – experiment – data set (keyword)
• Data set names (or part of it)
• Data set associated with codes
![Page 15: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/15.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 17
Hierarchical search for data sets
Project
Experiment
data set
keywords
![Page 16: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/16.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 18
Access to the data base
Access to the CERA data base:
• by java servlet - using an internet browser• by java applet (java runtime required) - obselet!• by java command line tool (jblob)
Unrestricted browsing for meta data
User account required for data download
Start: http://www.mad.zmaw.de/
![Page 17: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/17.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 19
Search for data (hierarchical)
![Page 18: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/18.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 20
Search for data (hierarchical)
![Page 19: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/19.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 21
Search for data (hierarchical)
![Page 20: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/20.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 22
Search for data (hierarchical)
List of data sets
Metadata output
Experiment decsription
![Page 21: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/21.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 23
Search for data (hierarchical)
List of data sets
![Page 22: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/22.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 24
Search for data (hierarchical)
Data set description
Parameter/Data organisation
![Page 23: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/23.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 25
Search for data (hierarchical)
Data set description:data organisation
![Page 24: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/24.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 26
Download data sets
x
Selecting data setfor download(login required)
![Page 25: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/25.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 27
Search data sets (by Code)
![Page 26: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/26.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 28
Search data sets (by Code)
Specifying code table
![Page 27: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/27.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 29
Search data sets (by Code)
Specifying code number
![Page 28: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/28.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 30
Search data sets (by Code)
Selecting data set
Add to process list
![Page 29: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/29.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 31
Search data sets (by dataset name)
![Page 30: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/30.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 32
Search data sets (by dataset name)
Specifyingname elemets
![Page 31: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/31.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 33
Search data sets (by dataset name)
Selecting data set
Add to process list
![Page 32: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/32.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 34
Download data sets
![Page 33: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/33.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 35
Download data sets
Process list:initiating data download
![Page 34: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/34.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 36
Download data sets
Specifying timerange for download
Time range
![Page 35: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/35.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 37
Download data sets
Specifying processing options
region
format
![Page 36: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/36.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 38
Download data sets
Startingdownload
![Page 37: The CERA2 Data Base Data input – Data output Hans Luthardt Model & Data/MPI-M, Hamburg Services and Facilities of DKRZ and Model & Data Hamburg, 29.3.2006.](https://reader036.fdocuments.in/reader036/viewer/2022062809/56649f335503460f94c50846/html5/thumbnails/37.jpg)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 39
Contact
URLs:
M&D : http://www.mad.zmaw.de/Data Base Portal : http://cera-www.dkrz.de/CERA/WDCC : http://www.wdc-climate.de
email: [email protected]