GCOOS AND GRIIDC: A Third Coast Partnership for Big Ocean...
Transcript of GCOOS AND GRIIDC: A Third Coast Partnership for Big Ocean...
GCOOS AND GRIIDC: A Third Coast Partnership for Big Ocean Data
Tools and Techniques for Data Transfers
M.K. Howard1, F.C. Gayanilo2, M Stössel1, and Steve Baum1
!1Dept. of Oceanography, Texas A&M University, College Station, TX, 77843, USA
2Harte Research Institute for Gulf of Mexico Studies, Texas A&M University-Corpus Christi, 6300 Ocean Drive, Unit 5869, Corpus Christi, Texas 78412, USA
2014 Ocean Sciences Meeting: 28-February 2014 Honolulu, Hawaii
Session: 113: Big Data for a Big Ocean: Progress in Data Availability and Interoperability
Gulf of Mexico Coastal Ocean Observing System
(GCOOS) !
Gulf of Mexico Research Initiative Information
and Data Cooperative (GRIIDC)
Authors and Outline
• GRIIDC - GCOOS!
• Large File Transport!
• DAP Transfers
Matt Nonong SteveMarion
GCOOS Holdings• Near real-time collection: 1500+ sensors!
• ~1.4M obs. per month. 2008-present!
• Historical: Ocean Data 1900-2000!
• Circulation Models & Model Resources!
• Pursuing: Coastal oxygen and nutrients!
• Future: Support ecosystem studies
Partial List of Models, Data, Instruments, and studies. !
Numerical Models: GCM: ROMS*, HYCOM*, FVCOM*, MITGCM, NCOM, ADCIR, Plume Dynamics: LES/Nek5000, Waves/Surf: UMWM, Delft3D, SWAN, Near-field: SMIP, CMS, HYDRO3D, Particle-tracking: LTRANS, Bay-Models: SUNTANS, Barataria Bay coupled model. (* multiple versions) Remotely-sensed Data: ENVISAT-SAR, MERIS, MODIS, JASON-SSH, LANDSAT, ASTER, AVHRR-SST, HF-RADAR Currents In-Situ: Autonomous: RAFOS, Surface/bottom/profiling drifters, drift-cards, dye, chemical tracers, AUV: video, multi-beam, Gliders: with temperature, salinity, velocity, chlorophyll, CDOM, acoustic backscatter, Personal water craft: shallow-water bathymetry, Expendables: AXBT, AXCP, AXCTD, GPS_Sondes, Optical: laser-altimetry, deep-LISST, CDOM, fluorescence, PAR, OBS, transmission, Video/Stills: polarimetric, infrared, sediment profile & traps, seabed, digital X-rays, Moorings: with conductivity, temperature, pressure, DO2, turbidity, currents, fluorescence, Other: water level, single-beam soundings, turbulence, Sediment samplers: box cores, Ekman grabs, dredges, benthic skimmers, traps, Sediment measurements: soil stress, cores, respiration, gas fluxes, light compensation., Chemicals: DOC, DIC, DON, DOP, NO3, NH4, UREA, PO4, SIO2, SRP, H2S, Eh, pH, C14, C13, Th234, Pb210, Hg-isotopes, methane concentration Selected Laboratory Studies: Concentration of compounds of crude oil in sediments, water, and tissue., Model of canyon flow, Core photography, resuspension rates of oil., Grain-size distribution-properties., Releases of crude and methane in a high-pressure environment in a laboratory setting., Surfactant effects on boundary layers., Physical processes of dispersion influencing droplet fate., Biodegradation over time and toxicity. (Amalgamated) in situ or in vivo biological studies: Bacteria: oil encounter rates, colony formation, films, oil interactions, and POC, DIC, DOC, N2 fixation rates, Microbial: community abundance, community characterization, identification, composition, process rates, degradation of oil, aerobic respiration, cell counts, transcription activity, and ecotoxcity of oil droplets, Macroinfauna: richness dominance, community structure, reproductive states, diversity, and gene expression, Mussel: growth, abundance, tissue response, hemocyte density, viability, and phagocytes, Zooplankton: concentration, abundance, distribution, composition, condition, grazing rates, and oil encounter rates, Phytoplankton: hydrocarbon signature, and dispersant toxicity, Fish: toxicology, reproductive potential, immune system function, stomach, muscle, gonad tissue, liver and bile, -exposure to PAH, growth rates, and diversity, Oil consumptions waste products (fecal matter and marine snow)., Insect species abundance, diversity, turn over and oil uptake, Birds: survival and nest success, fecal and stomach composition, and oil uptake. Marshes: Spartina stem number and diameter, Intercellular CO2 concentrations, measurements of above and belowground biomass.
Phys
ical
- C
hem
ical
Biol
ogic
alM
odelsField
LabField-Lab
Data Exchange Requirements• Move large files from provider to archive • Serve data subsets to consumers • Support humans and machine access
GCOOS Data Servers
• ERDDAP!• THREDDS Data Server (TDS)!• OPeNDAP
Where possible we encourage the use of: netCDF, COARDS/CF, NODC netCDF Feature Templates
Full Featured
To be deployed by GRIIDC in 2014-2015 timeframe.
Few Features
http://coastwatch.pfeg.noaa.gov/erddap/griddap/jplAmsreSstMon.htmlTable?tos[(2010-12-16T12:00:00Z):1:(2010-12-16T12:00:00Z)][(-89.5):1:(89.5)][(1.0):1:(360.0)],tosNobs[(2010-12-16T12:00:00Z):1:(2010-12-16T12:00:00Z)][(-89.5):
1:(89.5)][(1.0):1:(360.0)],tosStderr[(2010-12-16T12:00:00Z):1:(2010-12-16T12:00:00Z)][(-89.5):1:(89.5)][(1.0):1:(360.0)]
OPeNDAP
• Access remote data using URLs. • Remote subsetting via constraint expressions • Online browse -
• Data Set Descriptor Structure .dds (syntactic metatdata: shape 2D? 3D?)
• Data Attribute Structure .das (parameter names and units)
• Or combined .info
Tested Clients• MatLab (MathWorks) • IDL (Exelis) • IDV (UCAR) • EDC (NOAA/ASA) • Panoply (NASA) • GrADS (IGES/COLA) • Ferret (NOAA/PMEL)
Extractor
Viewer
Example Code
url = ‘http://urltodata/wrfout_0726_d01_2012-07-26.nc? LU_INDEX[0:1:0][0:10:810][0:10:855]' !ncid = ncdf_open(url) dataid = ncdf_varid(ncid,'LU_INDEX') ncdf_varget,ncid,dataid,lu_index
In ERDDAP: suffix is file type of response!In TDS: suffix is file type of source
Summary• GCOOS-GRIIDC: Strong Partnership!
• More data to appear in TDS/ERDDAP!
• Standards: Use them!
• RAM - buy lots!
• iPython/pyoos - yes!
Resources• http://www.opendap.org!• http://upwell.pfeg.noaa.gov/erddap/index.html!
• http://stommel.tamu.edu/~baum/bp.html#_pydap!
• http://stommel.tamu.edu/~baum/thredds2.html!
• http://stommel.tamu.edu/~baum/cages.html!
• http://www.giss.nasa.gov/tools/panoply/!
• http://www.unidata.ucar.edu/software/idv/!
• http://www.pfeg.noaa.gov/products/edc!
• http://www.mathworks.com!
• http://www.iges.org/grads/!
• http://www.ferret.noaa.gov/Ferret/!
• http://www.exelisvis.com/ProductsServices/IDL.aspx!