NMFS Enterprise Data Management 1 DAARWG Meeting December 8, 2010, 2010 Jim Sargent, NOAA Fisheries...

NMFS Enterprise Data Management


1

DAARWG MeetingDecember 8, 2010, 2010

Jim Sargent, NOAA FisheriesInformation Architect


The End in Mind: Take Aways

• Need to make quality data available to the world • It will take a cultural shift - Which is HARD!• NMFS EDM: One LO’s approach

• Understand the comprehensive enterprise-wide process we have been through, and

• Possibly leverage the fruits of our labors and learn from challenges • DAARWG recommendations:

– Expand scope beyond Access and Archive Requirements to full Data Management Life Cycle

– Promote and support EDMC and LO data management efforts– Leverage similarities while respecting diversity of data– Review and support EDMC’s Procedural Directives– Consider:

• Adopting a new vision of data management• Doing a comprehensive inventory


Outline

• A Case for a New Vision for Data and Data Management

• One LO’s approach: NMFS EDM

• Recommendations


• Volume and diversity of data• Collected for one-use; potential for multi-use• Infrastructure needed to support multiple access mechanisms• Releasability of data (organizational vs. technical) • Coordination of collection and distribution• Comparison and utilization of modeling outputs and observations

Overall Need for:• Clear and consistent data management policy• Better data documentation (metadata)• An overarching response plan

• Need to quickly identify applicable / available data

Deep Water Horizon Data Management Challenges

Prepared by Environmental Data Management Committee for NOAA Leadership

5

Pre and Post Disaster

Immediate Response


It’s the Next Disaster, Do We Even Know

What Data We Have?


“NOAA is like a library

without any card catalog

…or even bookshelves”

……..Dr. John A. Knauss, Former NOAA Administrator, 1986


• President's Directive on Open Government – Transparency, participation, and collaboration– Timely publication of quality information

• It’s the right thing to do!

• Greater than archive and preservation issues– Not all access is done from archives


Everyone is Wresting With This

• Other federal agencies – IWGDD, NSF, ICES, ISO

• DM maturing as a discipline– DAMA DM Book of Knowledge (DMBOK)– Certified Data Management Professionals

• H. R. 5037 April 15, 2010– Requires Federal agencies to develop public access

policies

• NMFS Science Center Data Management Reviews


It’s About a Cultural Shift• Need to move from the academic paradigm

– Publish or Parish Share or Parish– Science Science Information– Science Quality good enuf

• Changing Culture is Hard– Tipping Point, Blink

…. Malcolm Gladwell

– Switch, Made to Stick … Phil and Dan Heath


A New Vision??

NOAA data assets are recognized and managed as a core agency resource, on par with financial and human resources.

11


Science Board

Leadership

13

• Analysis & Research•Recommendations:

• Policies• Procedural directives• Guidelines• Best practices

• Implementation• Marketing

Approvals and support• Policies; • Procedural directives• Guidelines• Best practicesNMFS Staff

FIMAC

NMFS Staff

NMFS Enterprise Data Management14

2008 2009 2010

Jan 15, 2009IA Position

Filled

FIMC researched NMFS DM and developed

EDM Recommendations

• Recommendations presented to and accepted by LC

• FIMAC Created

•Teams Created• IMCs Created

FIMAC Workshop• Teams’ Plans developed• Required Resources Identified

Policy drafted and

vetted

Data Documentation

Procedure Directivedrafted and vetted

DataInventory Initiated

ImplementationPlanning and

Development of TeamsResearch, Analysis, and

Recommendations

A Brief History of EDM Time

Data Stewardship Teams Established

NMFS Data Catalog

established

Policy Enacted


15

NMFS EDM MISSION:

“To effect a cultural change in which all NMFS data are recognized and managed as a core agency resource, on par with financial and human resources.”


16

NMFS EDM Vision

NMFS customers

can confidently find, access,

and use our data


17

NMFS EDM Vision

NMFS customers


and use our data

Internal and external constituents


18

NMFS EDM Vision

NMFS customers


and use our data

Confidence in finding and trust in using our data


19

NMFS EDM Vision

NMFS customers


and use our data

• Using various portals, data.gov, geospatial.gov, etc) • Browse through an ordered hierarchies or taxonomonies• Search using:

• Discipline specific key words (controlled vocabularies)• User tags (folksonomies)• Metadata include users’ comments

• Minimize the number mouse clicks


20

NMFS EDM Vision

NMFS customers


and use our data

Through confidentiality and security filters while using standard tools and formats


21

NMFS EDM Vision

NMFS customers


and use our data

Download selected data withsufficient documentation, including quality indicators and warnings to effectively and properly use and understand the data


22

NMFS EDM Vision

NMFS customers


and use our data

All NMFS enterprise data we choose to share


23

NMFS EDM Vision

NMFS customers


and use our data


24

NMFS Enterprise Data And Information Management Policy

Overview

• Enacted by Eric Schwaab: June 2010– Directed RAs, SDs, and ODs to send 2-3 people to NMFS

Data Stewardship Workshop• General Policy: All data shall:

– be visible, accessible and understandable to authorized users;– modeled, named and defined consistently across and within

all NMFS programs;– have a standard set of metadata; and– be managed, controlled and shared by data stewards

throughout the data management lifecycle. – be publicly available generally within one-year of its collection

A high level NMFS Policy Directive implemented by Operational Procedure Directives


25

Procedural DirectivesPrinciples/Concepts

• Developing Procedural Directives an iterative process

• Data Documentation Procedural Directive: FMCs shall:– Develop their own plans for documenting and sharing data

assets– Inventory and document data assets and tools in NMFs

metadata repository, InPort– Measure metadata quality with a rubric

FY11 will be a year of learning and practicing how to document and share our data

• Develop, use, and refine metadata standards• Populate InPort with discovery level metadata• Develop practice and refine procedures for sharing


Data Stewardship Teams

26


NMFS Data Stewardship Workshop

• Discuss New Data Management Policies and procedures

• Develop best practices– How to document our data

• e.g., quality assessment

– Spirals in data documentation

– Rubrics and metrics

• Q&A and Sidebar Communities

27

To develop a shared understanding of data stewardship, empower the data stewardship teams to lead the way for implementing the Data

Documentation Procedure Directive, to develop best practice based on the experience during the initial documentation and inventory tasks.


Next Steps

• Data Doc. PD approved – Dec 2010

• Data Inventory Complete – Dec 2010

• Data Doc. Implementation Plans Due- Feb 2011

• Data Stewardship Workshop – Apr 2011

• First cut Best Practices – Apr 2011

• Data Doc. Implementation Plans Finalized – May 2011

• Address Preservation and Terminology

Socialize, Support, Promote, and Market


Critical Success Factors

• Management Commitment• Dedicated, Personally Committed Team

– FIMAC– Coordination Team– EDM Partners– Steady Hand at the Helm– Executive Sponsorship

• Drive Up and Then Drive Down– Harness the mavens

• Socialize, Support, Promote, and Market 29


Challenges

• Tipping Point almost reached but not there yet

• Budget Shortfall– Perception that field not fully bought into it

• Overworked Teams a high risk


DAARWG Recommendations• Expand scope beyond Access and Archive

Requirements to full Data Management life Cycle• Support cultural change needed to meet demand

for quality data• Promote and Support EDMC and LO DM Efforts• Review and Support Procedural Directives• Leverage similarities while understanding

diversity– Establish good data management practices for the whole

data management lifecycle• Consider Recommending

– A new Vision of Data Management – Conducting a comprehensive inventory with defined

metadata


32


33


Backup Slides

34


Data Sharing

• Data shall be shared in data.gov, as appropriate, as a one-click data asset, a one-click product, or by using a software tool (e.g., FOSS)

• Sufficient documentation to understand the data being shared must be published in InPort and referred to or provided with the data

• Data Stewards decide what data is appropriate for sharing

• Sharing confidential data must conform to Agency policy

“Data should be made as widely and freely available as possible while safeguarding the privacy of participants,

and protecting confidential and proprietary data”


37

Timelines for Sharing Types of DataData Asset

Type Description Sharing Time Frame

Continuous Time Series

Continuous data collections for the purpose of monitoring fisheries or environment

Annually, according to FMC Procedural directives implementation plan

Discrete Time Series

Surveys that collect data at discrete time intervals across months or years

Periodically based on survey design, according to FMC implementation plan

One-time collections

Results of experiments or small one-time collections

1 year or upon publishing of a work based on the data asset

Multiple Source

Compilation of data from multiple sources, e.g., compendiums syntheses, scientific reviews and studies

Upon publishing of a work based on the data asset, according to FMC implementation plan / publication policy

Derived Data*

Data and information developed using statistical and mathematical models and other technics.

1 year or upon publishing of a work or decision based on the data asset

Does not include provisional, predecisional documents and preliminary analyses leading to final management actions


1. Data should be archived and accessible

2. Adequate resources for end-to-end management

3. Management activities should involve users

4. Interagency and international partnerships

5. Metadata are essential

6. Expert stewards required for management

7. Process to decide what data to archive

8. Archive must support discovery, access, and integration

9. Effective management requires a formal, ongoing planning process

Prepared by Environmental Data Management Committee for NOAA Leadership38

National Research Council Committee on Archiving and Accessing Environmental and Geospatial Data at NOAA, 2007

Principles for Effective Environmental Data Management


39

Identifying the Problem

• Top priority Issues – Critical gaps – No authoritative data inventory– Insufficient metadata– Data quality and consistency challenges– Ability to integrate data– Administrative systems

• Other issues– Data being lost– Data not being archived for perpetuity– Historical data that need rescuing – Communications re: applications and IT

The FIMC identified 12 key issues and interviewed their management to determine their priorities

IssuesIdentification

Interestingly, the

lowest ranking

issue was that NMFS did not have buy-in

across FMCs for

addressing IMreturn


40

Identifying the Problem

• Top priority Issues – Critical gaps – No authoritative data inventory– Insufficient metadata– Data quality and consistency challenges– Ability to integrate data– Administrative systems

• Other issues– Data being lost– Data not being archived for perpetuity– Historical data that need rescuing – Communications re: applications and IT

The FIMC identified 12 key issues and interviewed their management to determine their priorities

IssuesIdentification

Interestingly, the

lowest ranking

issue was that NMFS did not have buy-in

across FMCs for

addressing IMreturn


NOAA’s Conceptual Framework

41


42

DAMA-DMBOK Functional Framework

NMFS Enterprise Data Management 1 DAARWG Meeting December 8, 2010, 2010 Jim Sargent, NOAA Fisheries...

Documents

Transcript of NMFS Enterprise Data Management 1 DAARWG Meeting December 8, 2010, 2010 Jim Sargent, NOAA Fisheries...