Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt...
-
Upload
daniella-francis -
Category
Documents
-
view
215 -
download
0
Transcript of Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt...
![Page 1: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/1.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Data Formats
Deniz Bozkurt
web.itu.edu.tr/bozkurtdITU Eurasia Institute of Earth Sciences
The International Training Course on“Climate Analysis and Applications”
Alanya10-19 October 2011
The International Training Course on“Climate Analysis and Applications”
Alanya10-19 October 2011
![Page 2: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/2.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Datasets
Climate Datasets
![Page 3: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/3.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Raw data
“abcd12345”
Input system
Digital representation “11010001”
Climate Datasets
![Page 4: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/4.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Datasets
• Modeled data: gridded multi-dimendisonal data– Consisting of variables, dimensions and attributes
• Observations (station, reanalysis)
![Page 5: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/5.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Datasets
• Climate data can be stored as different formats in different platforms:
• Character format
• Packed binary format
![Page 6: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/6.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate Data Formats - Character Format
• ASCII (American Standard Code for Information Interchange)
• It refers to a “text” file that is readable by the naked eye (it only contains the letters a-z, numbers, carriage returns, and punctuation marks.
• Each character is stored as digital
============
Binary100100011001011101100110110011011110101100010000011101111100111111001011011001100100
Hello, world
![Page 7: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/7.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ASCII
![Page 8: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/8.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ASCII
• Factors restrict the use of ASCII:
• Weak data storage and huge data size:
“9” 1 byte, “679.43” 6 byte, -0.123456E+05 13 byte
• No or not practical metadata
![Page 9: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/9.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – Packed Binary
• A common data format for saving and storing of large size of the data
• “3.1” 24 bits (ASCII) can be packed as 5 bit
• Can be processed very quickly and effectively in data analysis
![Page 10: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/10.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – Packed Binary
![Page 11: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/11.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Some standard data formats
• Self-describing data format– All the information about data is stored inside of the data
– No need for an additional information for the identification of the contents of the file
– Machine independent
• Can be readable by various applications– NCL, IDL, GrADS, Matlab, Ferret, C, C++, Java, Fortran
– Komut satırı operatörler: NCO, CDO
• Most commonly used format in the oceanographic and atmospheric science for observational data and numerical modeling (IPCC datasets, NASA, NOAA, NCAR, UCAR…)
NetCDF (network Common Data Form) Format
![Page 12: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/12.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
metadata
dimensions
variables
Variableproperties
attributes
![Page 13: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/13.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF format
• 3D (x,y,t) or 4D (x,y,z,t)
![Page 14: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/14.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF format
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
netcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
}
![Page 15: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/15.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF formatnetcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
Temperature = 111, 211, 311, 411;
}
141 241 341
131 231 331
121 221 321
441
431
421
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
time = 1
x = 1 to 4
y = 1
![Page 16: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/16.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF formatnetcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441;
}
time = 1
x = 1 to 4
y = 1 to 4
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
![Page 17: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/17.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
Data storage in NetCDF formatnetcdf mynetcdf {
dimesions:
x=4;
y=4;
time=UNLIMITED;
variables:
float x(x);
float y(y);
int time(time);
float temperature(time,x,y);
data:
x = 10, 20, 30, 40;
y = 110, 120, 130, 140;
time = 31, 59, 90;
Temperature = 111, 211, 311, 411, 121, 221, 321, 421, 131, 231, 331, 431, 141, 241, 341, 441, 112, 212, 312, 412, 122, 222, 322, 422, 132, 232, 332, 432, 142, 242, 342, 442, 113, 213, 313, 413, 123, 223, 323, 423, 133, 233, 433, 143, 243, 343, 443;
}
time = 1 to 3
x = 1 to 4
y = 1 to 4
Y
X
Time
![Page 18: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/18.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
4D (x,y,z,t)
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
Altitude
Climate data formats - NetCDF
![Page 19: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/19.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - NetCDF
some NetCDF commands
– ncdump file_name | less, to see the entire contents of the file
– ncdump –h file_name | less, to see the front portion of the data
– ncdump –v var_name file_name | less, to see the specific data within the file
– ncl_filedump to see the contents of different data formats
![Page 20: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/20.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - GRIB
GRIB (GRIdded Binary) Format
• World Meteorological (WMO) standard, commonly used in meteorology
• Bit-oriented data format: very efficient for transmission/archival
- netCDF (float) 2-4 times larger than corresponding GRIB
• two dimensional horizontal grids only
- each grid has scale/offset for optimal precision
• requires external parameter table (s)
![Page 21: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/21.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - GRIB
• Basically consists of:– Product definition section(PDS)
• Creater of the data, unit of the data, time stamp, vertical system of the data
– Grid description section (GDS), optional• Grid description, map projections
– Bit map section (BMS), optional• Defines which grid points will be presented
– Binary data section (BDS)• Contains the packed binary data
– End section
![Page 22: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/22.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats - GRIB
• Files are typically 1/2 to 1/3 of the size of normal binary files:
• Y: gridded data
• D: scale factor
• R: reference value
• E: binary scale factor
€
Y ∗10D = R + X ∗2E( )
![Page 23: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/23.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
• NetCDF data:• Raster
• Table
• Feature
• Directly readable
Climate data formats – ArcGIS Shapefile
![Page 24: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/24.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
![Page 25: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/25.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
Raster
Feature
Table
![Page 26: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/26.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
![Page 27: Eurasia Institute of Earth SciencesIstanbul Technical University Climate Data Formats Deniz Bozkurt web.itu.edu.tr/bozkurtd ITU Eurasia Institute of Earth.](https://reader036.fdocuments.in/reader036/viewer/2022062518/56649f1e5503460f94c3588a/html5/thumbnails/27.jpg)
Eurasia Institute of Earth SciencesIstanbul Technical University
Climate data formats – ArcGIS Shapefile
http://www.gisclimatechange.org/