1 Adventures in Web Services for Large Geophysical Datasets Joe Sirott PMEL/NOAA.
The NOAA National Geophysical Data Center
description
Transcript of The NOAA National Geophysical Data Center
![Page 1: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/1.jpg)
1
The NOAA National Geophysical Data Center And Collocated World Data
Service for Geophysics
Dan KowalData Administrator, Information Services Division
NOAA / NESDIS / [email protected]
GeoData Workshop 2014
Failure to Connect?
![Page 2: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/2.jpg)
Technical issues of connecting geodata in and between governmental agencies.
![Page 3: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/3.jpg)
Challenges and Accomplishments
• Metadata Publication• Software Development• Data Citation
![Page 4: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/4.jpg)
Metadata Tools
http://www.ngdc.noaa.gov/docucomp/
![Page 5: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/5.jpg)
Measurement of Completeness
Records Rubric Scores
Valid Invalid Count ≥ 20 Count ≥ 25 Mean Min Max
3314 218 3157 2512 22.9 6 41
![Page 6: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/6.jpg)
Count of Broken URLS
Components Other Xlinks Broken URLs Broken Xlinks
Count Reuse Count Reuse Count Reuse Count Reuse
277 70570 3 133 34 202 22 226
![Page 7: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/7.jpg)
Metadata Publication - Local• NGDC Metadata H
omepage– Immediately
available
• NGDC Geoportal – synchronized
weekly or upon request
![Page 8: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/8.jpg)
![Page 9: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/9.jpg)
Software Challenges● Wide variety of data types● Diversity of data providers● Decreasing staff and funds● Increasing number of data sets ~ 600 to
date● Legacy code bases● Lack of communication
![Page 10: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/10.jpg)
Engineering Objectives● Common framework
o standardize on common technologies, shared knowledge, centralization supporting tracking / reporting
● Isolate dataset specific componentso share things like file handling, messaging across
disparate datasets● Modular and extensible
o ease maintenance and facilitate testing, phasing in new capabilities (incremental improvements), reduce likelihood of system-wide impacts to errors or malfunctions
![Page 11: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/11.jpg)
Engineering Objectives - cont’d
● Industry-standard and best practices and patternso develop in teams, automated builds, test
coverage, leverage industry tools● Resilient
o eliminate single points of failure, be able to restart processes following errors without data loss, secure
● Minimize custom codeo reduce software maintenance
![Page 12: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/12.jpg)
12
New Access Interfaces at NGDC
![Page 13: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/13.jpg)
DOI Landing Page
13
![Page 14: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/14.jpg)
14
DOI Landing Page
![Page 15: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/15.jpg)
DOI Readiness Assessment
![Page 16: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/16.jpg)
Data Citation Summary• Data Linkage to Publications:
– Data Citation Index in Thomson-Reuters’ Web of Knowledge– Elsevier ScienceDirect – Ongoing discussions.
• Procedural Directive for Data Citation in the works. – Leverage ESIP Guidance– NCAR’s Data Citation White Paper
• DataCite – ~ 50 Datasets minted through EZID.
![Page 17: The NOAA National Geophysical Data Center](https://reader035.fdocuments.in/reader035/viewer/2022081505/56815f68550346895dce6a9e/html5/thumbnails/17.jpg)
In Summary…
• Need to fix the catalog publishing disconnect.• Enterprise approach to development paying dividends.– Creating opportunities for reuse.– Generic functionality shared across data sets.– Going to take more resources to transition legacy data sets.
• Collaboration in Data Citation practices across Data Centers bodes well for future consolidation.
• Begin “Interoperability” discussion early when initiating a new Archive Project.