June 23-24, 2011. Development of a NOAA Policy on the Use of External Data Sources 2 DAARWG...

Scott HausmanDeputy DirectorNational Climatic Data Center

Development of a NOAA Policy on the use of

External Data Sources

Data Archive and Access Requirements Working Group (DAARWG) Meeting

June 23-24, 2011

Development of a NOAA Policy on the Use of External Data Sources

2DAARWG Meeting June 23-24, 2011

Overview• Motivation for Data Sharing Partnerships

• Proposed Climate Service• Interagency Climate Portal• National Climate Assessment

• Data Management Challenges of External Data• Example: CoCoRaHS• Example: EarthNetworks GHG Network

• Existing NOAA Policy• Use Case: Argo Ocean Floats



Proposed Climate Service

“The Climate Service Line Office at NOAA … would work with the broader climate service enterprise … to provide businesses, communities, and resource managers with services and information for decision-making.

The proposed Climate Service Line Office at NOAA would improve NOAA’s organization, such that the agency can be a more accessible, transparent, and collaborative partner…

NOAA’s reorganization would also support economic innovation and entrepreneurship. This includes supporting development of the private sector climate services industry.”

— Written Statement to the House Committee on Science, Space and TechnologyUnder Secretary Jane Lubchenco, June 22, 2011

Partnership a Key Tenet for Success



Proposed Climate Service

Information Delivery and Decision SupportNOAA uses its national and regional infrastructure to deliver climate services today

Assessments of Climate Change and Impacts NOAA is a leader in national and regional climate impact assessments Over 70% of Federal IPCC AR4 WG1 authors were from NOAA

Climate Change Research and ModelingInternational award winning models of the global climate

Climate Observations and MonitoringNOAA operates over 90 observation and monitoring systemsNOAA is mandated to monitor and provide access to climate data and information

Security

Forestry

Water

Health

Infrastructure

LandManagement

Oceans

Energy

Federal Response to the nation’s

climate challenges

NOAA’s AssetsPartnerships & Collaboration

Vision andStrategic

Framework



Interagency Climate Portal

• National Climate Adaptation Summit, called for by John Holdren, Director of the White House Office of Science and Technology Policy, issued “seven priorities for near-term action” in 2010, including

“Creating a federal climate information portal. This would provide single-point access to data from all relevant federal agencies and programs and would evolve over time into a more ‘national’ portal with information about relevant non-federal efforts.”

• Also, Executive Office of the President, Committee on Environment, Natural Resources, and Sustainability (CENRS) Roundtable on Climate Information Services called for

an interagency climate information portal plan, focusing on National Climate Assessment

Administrations Call for Action



Interagency Climate Portal

• A National Climate Portal as envisioned is an immense task

• Use the National Climate Assessment (NCA) to focus the initial scope

• The NCA can be…• A starting point, extensible to interagency data• A driver of requirements for Data Management Planning:

transparency, access, usability, data quality, and business rules

• A test case for connecting robust and federated data structures with appropriate user experiences

Where to begin development?



National Climate Assessment

• A product AND a sustained process• More stakeholder engagement and contributions• More transparency, traceability, accessibility and usability• More data, more authors, more users• A suite of Indicators (discrete and composite, physical and societal)• Next report due in 2013, draft due in 2012, Federal Advisory

Committee just formed• Snapshots plus frequent updates

• More frequent interim reports, updated indicators and fresh web-content are expected… BUT

• An assessment is necessarily a snapshot in time and should be reproducible

A Change in Direction from Past Reports




• External Data• Clear need to move beyond government data and the

peer reviewed scientific literature as information source, but who guarantees access and a continuous archive?

• Quality• Information Quality Act requirements (highest level)• Peer review requirements (highest level)

Data Management Planning Consideration




• NCA will likely not ‘host’ basic data, but will provide central access to federated system (critical questions remain for external data)• Highly-processed, derived data used in the report may be centrally hosted

• Must accommodate huge data sets• Too big to move, too big to host all data in one location• Implies distributed data architecture, federated storage

• Metadata must adhere to common standards• Uncertainty must be documented• Quality and original source of data must be clear and peer reviewed• Data system must be interoperable

• Use of data.gov must have clear guidelines

• Narrative assessment and graphics must link to source data and information

Data Management Requirements



External Data Management Challenges

• Data Documentation and Quality• Are the metadata sufficient and standardized?• Does the data come from a trusted source (NASA vs. Joe’s Data

Shack) and what is the process to certify data sources as “trusted”?• How robust are the science algorithms and quality control

procedures?• Data Access and Usage

• Are the data accessible via standardized Web services?• Is the data source operationally reliable?• Are there data usage or cost restrictions?

• Data Retention• What are the long-term archive prospects for the data?• Must NOAA ensure the data are archived?• Can NOAA depend on the data whenever necessary?



Examples of External Data Sources

• Unique, non-profit, community-based network of volunteers working together to measure and map precipitation (rain, hail and snow)• Using low-cost measurement tools, stressing training and

education, and utilizing an interactive Web-site, aim is to provide the highest quality data for natural resource, education and research applications

• NWS routinely uses CoCoRaHS data and NCDC includes in snow archives

• Non-standard procedures along with data quality and continuity issues have affected the climate record• FEMA relies on NOAA-provided snow totals to assess natural

disasters

• Snow Workshop, May 25-27, 2011organized to address these and other issues

Community Collaborative Rain, Hail & Snow Network (CoCoRaHS)



Examples of External Data Sources

• Over the next five years, Earth Networks willinvest $25 million to build and deploy a state-of-the-art greenhouse gas emissions observation and measurement network

• By the middle of 2012, Earth Networks will deploy 100 greenhouse gas measuring instruments • 50 in the United States• 25 throughout Europe• 25 elsewhere around globe

• Collaborating with Scripps Institution of Oceanography

• Potential contributor to NOAAcarbon observing network?

Earth Networks GHG Observing Network



Existing NOAA Policies

• Much of data NOAA receives from external sources is in the form of raw observations, or “original data”

• IQA requires that original data be managed using documented processes with routine QC/QA

• http://www.cio.noaa.gov/Policy_Programs/info_quality.html

NOAA Information and Quality Act (IQA) Guidelines

http://www.cio.noaa.gov/Policy_Programs/info_quality.html

http://www.cio.noaa.gov/Policy_Programs/info_quality.html




• Section 3.04. …NOAA will take advantage of existing capabilities and services of commercial and academic sectors to support efficient performance of NOAA's mission and avoid duplication and competition in areas not related to the NOAA mission…

• …NOAA will give due consideration to these abilities and consider the effects of its decisions on the activities of these entities, in accordance with its responsibilities as an agency of the U.S. Government, to serve the public interest and advance the nation's environmental information enterprise as a whole.

• http://www.corporateservices.noaa.gov/~ames/NAOs/Chap_216/216-112.pdf

Policy on Partnerships in the Provision of Env Info

http://www.corporateservices.noaa.gov/~ames/NAOs/Chap_216/216-112.pdf

http://www.corporateservices.noaa.gov/~ames/NAOs/Chap_216/216-112.pdf




• Primarily directs NOAA’s provision of data to the public.• Data layers provided by sources other than federal agencies…and

displayed on NOAA Web pages must be used judiciously, because these situations present special challenges with respect to maintaining agency Information Quality (IQ) Act standards. Such data layers must meet NOAA IQ requirements for NOAA use of third-party information… In addition, any external data from third-party providers must meet the following criteria: • The data must be must be necessary for, and material to, the presentation of

agency information or the delivery of agency services, and the map must credit the contributing source of the data or provide a direct link back to the third-party source data provider.

• The data must be relevant and timely, and complete steps must have been taken to ensure that data layers are actively updated to achieve the highest level of quality possible.

• http://www.cio.noaa.gov/Policy_Programs/NOAA_web_mapping_applications_policy.html

Web Mapping Applicaton Policy Implementation Guide

http://www.cio.noaa.gov/Policy_Programs/NOAA_web_mapping_applications_policy.html

http://www.cio.noaa.gov/Policy_Programs/NOAA_web_mapping_applications_policy.html



Use Case: Argo Ocean Floats• Provider: International - Multiple Nations• Trusted Source(s): Yes (Global Ocean Observing

System)• Purpose: Long-term climate monitoring of sea surface

temperature and salinity• Science Quality: Excellent (e.g., Scripps Institution of

Oceanography)• Metadata: Excellent• Data Access: Excellent with multiple access points• Restrictions on the use of the data: None• Long-term Archive Dependability: Excellent



Summary• NOAA is committed to partnering with other data

producers, even more so in dealing with the challenges of the National Climate Assessment

• NOAA has been using external data sources for a long time for a number of applications, and this will continue

• An overall NOAA policy documenting this usage (taking into account existing policies across the agency) is needed to inform acquisition decisions

• NOAA has an opportunity to lead the way in defining data management practices more broadly

• The DAARWG is well positioned to advice NOAA on the content of such a policy



Conclusion• Task is to develop a NOAA policy that defines

what data should be acquired through non-NOAA sources and what standards should be applied to such acquisitions

• Agenda• National Weather Service Existing Policy (Ed Johnson)• IOOS and Data Provider Certification (Zdenka Willis)• National Climate Assessment Consideration (Anne

Waple)

?Scott HausmanDeputy DirectorNOAA’s National Climatic Data Center (NCDC)151 Patton Avenue, Room 557Asheville, NC 28807-5002 828-271-4848 828-271-4246 828-450-9188

[email protected]

Thank you!

June 23-24, 2011. Development of a NOAA Policy on the Use of External Data Sources 2 DAARWG...

Documents

Transcript of June 23-24, 2011. Development of a NOAA Policy on the Use of External Data Sources 2 DAARWG...