Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007...
-
Upload
clyde-parsons -
Category
Documents
-
view
213 -
download
0
Transcript of Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007...
![Page 1: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/1.jpg)
Geoscience Data Analysis and Visualization
Tools from NCAR
HDF/HDF-EOS Workshop XI
Nov 6-8, 2007David Brown
![Page 2: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/2.jpg)
HDF/HDEOS Workshop XI, November 2007
Topics• Two interfaces: the same capabilities (mostly)
– NCL - a self-contained scripting language– PyNGL and PyNIO - Python modules
• Quick overview: visualization and analysis• Data model • State of our I/O component development• Enabling access to HDF 5, HDFEOS 5, and
NetCDF 4
![Page 3: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/3.jpg)
Brief HistoryBrief HistoryNCAR NCAR
GraphicsGraphicsHLUsHLUsNCLNCLNIONIO
PyNGLPyNGLPyNIOPyNIO
PyNIO and PyNGLJuly 2006
(Numeric 24.x/NumPy 1.0)
PyNIO2005
PyHLU2003
PyNGL 2004
Original FortranLate 60s
GKS/CGM/Fortran 77/UNIX conversion1980s NCAR
Graphics
1992: NCARG ConferenceC interface
Mid 90sHLUs
Mid 90s
NCLMid 90s
GSUN - Late 90s
1995: NCL Conference
2000-present: 30+ NCL Workshops
NIO2005
![Page 4: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/4.jpg)
HDF/HDEOS Workshop XI, November 2007
NCAR Command Language (NCL)A scripting language designed for
scientific data analysis and visualization
1. Simple, robust file input and output
2. Hundreds of analysis functions
3. Visualizations (2D) are world class and highly customizable
http://www.ncl.ucar.edu/
![Page 5: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/5.jpg)
HDF/HDEOS Workshop XI, November 2007
NCL: Data analysis
• Array-based syntax and operators• Hundreds of functions- Spherical harmonics- Scalar and vector regridding- Vertical interpolation- EOFs
• Many tailored to geosciences• Most handle missing data• Can call C and Fortran routines
![Page 6: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/6.jpg)
HDF/HDEOS Workshop XI, November 2007
NCL: Visualization
• High-quality and customizable visualizations• Contours, XY, vectors, streamlines• Maps with most common map projections• Handles data on regular and irregular grids,
triangular meshes• Specialized scripts for skew-T, wind roses,
histograms, Taylor diagrams, panels, bar charts• GSUN interface: simplifies visualization • Over 1,400 visualization options available
![Page 7: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/7.jpg)
HDF/HDEOS Workshop XI, November 2007
![Page 8: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/8.jpg)
HDF/HDEOS Workshop XI, November 2007
PyNGL - Python module• Python NCL Graphics Library • Python version of popular GSUN interface• Same publication-quality graphics as NCL• Utilizes existing Python modules and
development tools (swig, Numeric, NumPy)• Contains some climate-specific data
analysis functions• Extensive and updated documentation
http://www.pyngl.ucar.edu/
![Page 9: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/9.jpg)
HDF/HDEOS Workshop XI, November 2007
NCL Data Model
• Based on netCDF 3 • Language variables can have:• Named dimensions• Attributes• Coordinates variables
• A convention in NetCDF but a language feature in NCL
• 1d array with the same name as a dimension• Contains coordinate values for dimension elements
![Page 10: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/10.jpg)
HDF/HDEOS Workshop XI, November 2007
Supported file format I/O• One function reads all supported data
formats: • NetCDF (now including NetCDF4 classic),
HDF4, HDF-EOS 2, GRIB 1 and 2• Writes NetCDF and HDF4
• All file formats massaged into same model: file variables have basically the same features as internal variables
• Flat name space: HDF group names are appended to variable names
![Page 11: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/11.jpg)
HDF/HDEOS Workshop XI, November 2007
PyNIO - Python module• Same I/O library as NCL (libnio)• Reads and writes same formats as NCL• Same NetCDF-like view of all formats• But interfaces with NumPy for data access• Modeled on the Scientific NetCDF module
– NioVariable type is a reference to the variable in the file (metadata is attached to variable)
– Dereferencing it results in a NumPy variable with no metadata
• *Numpy is a Python array processing module
http://www.pyngl.ucar.edu/Nio.shtml
![Page 12: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/12.jpg)
HDF/HDEOS Workshop XI, November 2007
2D Coordinates• Traditional NetCDF coordinate variables are
1D vectors• However, satellite data and model data on
modern grids require 2D coordinate grids• 2D coordinate variables created on the fly for
GRIB grids and HDFEOS Grid-type data• Added value variables that allow data to be
plotted on any map projection
![Page 13: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/13.jpg)
HDF/HDEOS Workshop XI, November 2007
Enabling NetCDF 4 ‘classic’ • Recompile with NetCDF 4 and HDF 5 1.8
beta (we also included szip)• Add options for specifying format and
compression level• Tests using ncl_convert2nc on GRIB files
show reduction in file size by ~1/2 over GRIB.• Caveats:
– Beta version NetCDF 4 not yet supported on some architectures (64 bit mainly)
– Not supported for OPeNDAP NetCDF client library
![Page 14: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/14.jpg)
HDF/HDEOS Workshop XI, November 2007
Path to HDF, HDFEOS 5 and full NetCDF 4 support
• size_t for dimension sizes to enable large variables on 64-bit hosts
• Add support for more atomic types (int64) • Support for groups in the file context• User control of chunking• Compound and variable length data types• Components to read HDF 5 and HDFEOS 5• Current NetCDF module extended to handle v4• More support for aggregation
![Page 15: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/15.jpg)
HDF/HDEOS Workshop XI, November 2007
Current status• NCL 5.0.0 released this week
– NCSA-style Open Source license (finally)– Binaries available under slightly more restrictive
license (because of included 3rd party software)
• PyNGL/PyNIO 1.2.0 available– Supports NumPy 1.0.x and Python 2.5– Binaries (easy installation) available for:
• Various flavors of Linux, Mac OSX, Cygwin
• Other Unix systems, 32 and 64 bit
![Page 16: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/16.jpg)
Distribution of PyNGL/PyNIO sites
Distribution of NCL sites
![Page 17: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/17.jpg)
HDF/HDEOS Workshop XI, November 2007
Support
• Websites with extensive documentation
• Tutorials
• Hundreds of examples with downloadable scripts
• Active email lists– [email protected], [email protected]
• Hands-on training workshops
![Page 18: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/18.jpg)
HDF/HDEOS Workshop XI, November 2007
Download locations
• NCL
• PyNGL and PyNIO
http://www.pyngl.ucar.edu/download.html
http://www.ncl.ucar.edu/download.html
![Page 20: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/20.jpg)
HDF/HDEOS Workshop XI, November 2007
Resources (a.k.a. attributes)
• Same control mechanism for NCL and PyNGL
• Detailed control of the appearance of a visualization
• Example using POP grid ocean current data
![Page 21: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/21.jpg)
HDF/HDEOS Workshop XI, November 2007
![Page 22: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/22.jpg)
HDF/HDEOS Workshop XI, November 2007
![Page 23: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/23.jpg)
HDF/HDEOS Workshop XI, November 2007
![Page 24: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/24.jpg)
HDF/HDEOS Workshop XI, November 2007
![Page 25: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/25.jpg)
HDF/HDEOS Workshop XI, November 2007
![Page 26: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/26.jpg)
HDF/HDEOS Workshop XI, November 2007
NCL and PyNGL visualization examples
• scatter and line plots, contours, vectors, skew-t, meteograms, maps, wind barbs, bar charts, streamlines, trajectories, filled polygons, paneled visualizations
![Page 27: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/27.jpg)
![Page 28: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/28.jpg)
![Page 29: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/29.jpg)
Skew-T graphicCourtesy ofDennis Shea
A Skew-T plot is used by
meteorologists to analyze data from
a balloon
sounding.
![Page 30: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/30.jpg)
![Page 31: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/31.jpg)
![Page 32: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/32.jpg)
![Page 33: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/33.jpg)
Based on a visualizationof Adam Phillips
![Page 34: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/34.jpg)
Based on a visualizationof Joel Norris (Scripps)
using dummy data
![Page 35: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/35.jpg)
Data from Climate Analysis SectionChristian Guillemot
![Page 36: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/36.jpg)
THORPEX - an international research and development program responding to the weather related challenges of
the 21st century to accelerate improvements in the accuracy of 1-day to 2- week high impact weather
forecasts for the benefit of society, the economy and the
environment.
GRIB 2 TIGGE (THORPEX Interactive Grand Global Ensemble) data
Japan Meteorological Agency
![Page 37: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/37.jpg)
Courtesy Jeff YinNCAR CGD
![Page 38: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/38.jpg)
Image courtesy of Nan Rosenbloom, CGD
![Page 39: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/39.jpg)
First two map databases built-in; high-resolution available as simple download
![Page 40: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/40.jpg)
Christophe Cassou (CNRS/CERFACS)
This grid could be described as a tripole grid that is further modified by the arbitrary displacement of some portions of the grid to achieve finer resolution over areas of interest (typically, ocean areas).
![Page 41: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/41.jpg)
Courtesy Mark Stevens, NCAR CGD
![Page 42: Geoscience Data Analysis and Visualization Tools from NCAR HDF/HDF-EOS Workshop XI Nov 6-8, 2007 David Brown.](https://reader036.fdocuments.in/reader036/viewer/2022062802/56649ed95503460f94be7b45/html5/thumbnails/42.jpg)
Grid from Tom GrossNOAA/NOS/CSDL/MMAP