Requirements and lessons learned from operations of Data Services for Antarctic Science ·...

16
www.iedadata.org Metadata and Data Curation Requirements and lessons learned from operations of Data Services for Antarctic Science S. Carbotte Sept 10, 2013

Transcript of Requirements and lessons learned from operations of Data Services for Antarctic Science ·...

Page 1: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

www.iedadata.org

Metadata and Data Curation

Requirements and lessons learned from operations of Data Services for Antarctic

Science

S. Carbotte Sept 10, 2013

Page 2: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Metadata essential for all aspects of data curation

• Many kinds of metadata: Administrative (provenance, credit), descriptive (supports discovery and identification), technical, structural and preservation metadata (DCC – UK)

• Metadata needs of data creators can be very different from those of data curators and re-users

• Define metadata in relation to it’s intended use (e.g. for searching, display, science re-use, long term

preservation etc).

Page 3: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

MetaData requirements from Science User Perspective

• Primary desired search capability – Map based & data type (&temporal)

• Want to find all data types in search area

- multidisciplinary research

• Want to identify only data of interest

• Documentation of data quality

• Primary publications for further information

• Enough metadata to track provenance and ensure credit

Page 4: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

1.Antarctic and Southern Ocean Data Portal 2. USAP Data Coordination Center 3. SESAR sample catalog

http://www.marine-geo.org/portals/antarctic

Antarctic Data Services:

IEDA

http://www.usap-data.org/

Page 5: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Documentation registration and preservation of all USAP marine data

• Search and Download

• Investigator Support

• Data Analysis& Visualization www.marine-geo.org/portals/antarctic

ASODS

Page 6: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Diverse marine data types: Bathymetry, Multi-channel Seismic, ADCP, CTD, Salinity, Side-scan, Imagery, Turbidity…

• Field and Derived data (layer thickness, geologic interps, microseism catalogs…)

• Multiple File Formats: NetCDF, SEGY/SEGD, MB, JPEG/GIFF, ASCII (many)

• Data Holdings: 270,000 files, 2.8 TB, history of R/V Palmer and Gould expeditions

Page 7: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

Sample-Based Services SESAR, PetDB, SedDB

IEDA

Page 8: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• All NSF funded research

documented within

GCMD/AMD

• Signatories of Antarctic treaty

submit

• USAP-DCC Assists ANT

investigators with NSF data

obligations, DIF creation

&data submission

• Tools for data discovery and

access including web services

US Antarctic Program Data Coordination Center

Page 9: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Metadata supports diverse Data Access options: Search Portal, Client Viz and Analysis Applications, Web Services

Page 10: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Metadata supports data publication, domain science re-use, education, data preservation

Page 11: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Data contribution from scientists with comprehensive metadata for full range of needs is challenging

• Standards do not exist for many data types- disciplinary data centers must be involved in developing standards

• New requirements arise from different stakeholders (e.g. NSF, CI community)

• Active use of repository is essential for quality of metadata

Lessons Learned – Metadata

Page 12: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

• Work directly with instrument/facilities operators where ever possible/applicable- PIs are busy!

• Harvest existing resources for metadata where possible

• Need tools for metadata collection in the field

• Annotation tools needed for scientists to add to and correct metadata

Lessons Learned – Metadata

Page 13: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

CI enables research and is rapidly evolving

Existence of disciplinary and backbone databases are essential to take advantage of these developments

Data holdings need to be accessible via programmatic interfaces

Data systems need to adapt/evolve - must be agile (difficult given funding structures)

Lessons Learned - Operating Principles

Page 14: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

Incentives for scientists to contribute are essential

Funding agency sticks –not so effective

Attribution is essential

CI tools needed to improve scientists ability to visualize and analyze their own data and other multi-disciplinary data (beyond discovery)

Data Publication – attribution and credit

Lessons Learned - Operating Principles

Page 15: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

To build a system that will be used requires close interaction with Science Users (data creators and consumers)

Custom interfaces for user communities may be needed

Quality of system depends on usage (errors and weakness revealed through use)

Value of system increases with longevity and growth in data holdings - requires contribution and engagement of user community and sustained funding commitments.

Lessons Learned – Operating Principles

Page 16: Requirements and lessons learned from operations of Data Services for Antarctic Science · 2018-07-18 · Requirements and lessons learned from operations of Data Services for Antarctic

Lessons learned after 190M lessons served

Lessons learned after 190M lessons served

Detecting Financial Reporting Fraud – Lessons learned · PDF fileDetecting Financial Reporting Fraud – Lessons learned by ... Detecting Financial Reporting Fraud – Lessons learned

Detecting Financial Reporting Fraud – Lessons learned · PDF fileDetecting Financial Reporting Fraud – Lessons learned by ... Detecting Financial Reporting Fraud – Lessons learned

Lessons Learned:

Lessons Learned:

Lessons learneD

Lessons learneD

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lessons learned & not learned at MSU

Lessons learned & not learned at MSU

Lessons learned

Lessons learned

The Apollo Experience Lessons Learned for Constellation ... · PDF fileThe Apollo Experience Lessons Learned for Constellation Lunar Dust Management ... Experience Lessons Learned

The Apollo Experience Lessons Learned for Constellation ... · PDF fileThe Apollo Experience Lessons Learned for Constellation Lunar Dust Management ... Experience Lessons Learned

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Lessons Learned fromLessons Learned from Structural ... learned davy beicker.pdf · Lessons Learned fromLessons Learned from Structural Engggineers Learning ... reduced size of the

Lessons Learned fromLessons Learned from Structural ... learned davy beicker.pdf · Lessons Learned fromLessons Learned from Structural Engggineers Learning ... reduced size of the

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned

NSDI Lessons learned

NSDI Lessons learned

DDM UPDATE LESSONS LEARNED & LESSONS SHARED …

DDM UPDATE LESSONS LEARNED & LESSONS SHARED …

D530.1 - Lessons Learned Framework Concept€¦ · Driving Innovation in Crisis Management for European Resilience D530.1 - Lessons Learned Framework Concept Keywords: Lessons learned,

D530.1 - Lessons Learned Framework Concept€¦ · Driving Innovation in Crisis Management for European Resilience D530.1 - Lessons Learned Framework Concept Keywords: Lessons learned,

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

Lessons learned writing exploits LESSONS LEARNED WRITING EXPLOITS Gerardo Richarte Iván Arce

Lessons learned writing exploits LESSONS LEARNED WRITING EXPLOITS Gerardo Richarte Iván Arce

Lessons learned from Asian tsunami disaster: Sharing knowledgeusir.salford.ac.uk/9805/1/lessons.pdf · Title Lessons learned from Asian tsunami disaster: ... Lessons learned from

Lessons learned from Asian tsunami disaster: Sharing knowledgeusir.salford.ac.uk/9805/1/lessons.pdf · Title Lessons learned from Asian tsunami disaster: ... Lessons learned from