DOI Climate Change Task Force CC Impacts Data Collection, Integration, and Management Workgroup...

DOI Climate Change Task Force

CC Impacts Data Collection, Integration, and Management

Workgroup

Recommendations of the Data Management/Integration

Subcommittee

Steve FancyNational Park ServiceInventory and Monitoring Program Leader

Chaco Culture Natl. Historical Park

The climate change crisis is pushing agencies to work together in ways that we’ve never been able to do before.

“A crisis is a terrible thing to waste”

“Research funding agencies need to recognize that preservation of and access to digital data are central to their mission, and need to be supported accordingly” Nature: 10 Sep 2009

Climate Change Response

• Data and information sharing and integration are the key to collaboration among DOI bureaus and their many partners in addressing the significant climate change challenges.

• We need a significant DOI response to develop effective and efficient means of finding, retrieving, using, and sharing the best available data and information. This will require both a top-down and bottom-up approach to addressing Department-wide data management issues.

• The key to data sharing and integration is modernization of data systems using reusable and sharable building blocks (“web services”) based on DOI and industry standards to allow data exchange across multiple data systems.

Key Points from DOI Data Management/Integration Subcommittee:

Service-oriented Architecture

Service-oriented architecture allows agencies to maintain their internal systems, yet make portions of their data available for sharing

by using a common set of standards and processes.

NPS

FWS

USGS

NASA

NOAA

States

Internal data

Shared dataEnd users

NPSNPS

ResearchersResearchers

StatesStates

NOAA

General PublicGeneral Public

FWS

Natural Resource

Portal

NPS

FWS

USGS

NASA

NOAA

Air France

Lufthansa

DeltaFrontier USAir

United Alamo

Hertz

NationalBudget Enterprise

Avis Best Western

Days Inn

HiltonMarriott Motel 6

Holiday Inn

Expedia.comOrbitz.com

Common Service Standards Common Service Standards Common Service Standards

Hotels.com

Using a set of common data service exchange standards makes disparate data available for sharing

Recovery PlansSpecies ReportsSpecies StatusMigratory BirdsRefuge LocationsCritical HabitatWetlandsEtc.

ReferencesTaxonomySpecies OccurrencesPark LocationsMonitoring DataWater QualityVegetationEtc.

PublicationsPhenologyRemote SensingStream GaugesClimate StationsExotic SpeciesITIS, PLANTSMaps, etc.

Data.gov

DOI

Common Service Standards Common Service Standards Common Service Standards

Using a set of common data service exchange standards makes disparate data available for sharing

FWS Data

NPS Data

Natural Resource Portal

USGS, Other Agencies

Researchers

• Integration of Resource Management Applications

• Service-oriented Architecture (SOA): build once, use many times services that can be re-used by many applications

• Using DOI and industry standards to allow data exchange and integration among data systems

• DOI award for “Best Agency SOA Application”

Why Service-oriented Architecture?• Modular: Each agency/organization can build their own system

incrementally and iteratively, when they’re able. Agencies don’t all have to get together and “figure it all out” in advance.

• SOA can leverage existing investments; more efficient– Capitalize on existing resources, including developers, software

languages, hardware platforms, databases, and applications– “Reusable building blocks” developed by others can be shared

and built upon, rather than everyone starting from scratch.

• Adaptive Management for IT: SOA positions us to respond rapidly to a changing environment once initial building blocks are in place.

• SOA is DOI, industry, and Federal Government Best Practice– Private sector investment in development of methods.

Service-oriented ArchitectureSOA: silos are deconstructed into reusable, multipurpose “services”

user identitypark codereferencesdata sets

access control data entry

data downloadsreports

observationstaxonomyvouchers


access control data entry

data downloadsreports

observationstaxonomyvouchers


access controldata entry

data downloadsdata uploads

reports




reports

Species Data

References




reports




reports

Data Store

Silos: stand-alone systems that duplicate data and don’t interact well

User identityservice

User identityservice

Unit ServiceUnit Service

Reference Service

Reference Service

Taxonomy Service

Taxonomy Service

SpeciesService

SpeciesService

NotificationService

NotificationService

ATBI All-TaxaInventories

ResearchPermit

And Reporting

Researchers

States

NOAA

University Cooperators

Legacy Systems New Integrated System

APCAM InvasiveSpecies

External users such as other agencies, universities, or organizations can access services we want to share

Internally, services can be shared and re-used

USFWS

Examples of common, reusable services

References(Metadata)

Taxonomy

Identity Management

Notification

Feedback

Fluency in and crosswalks among key taxonomic systems. Allows analysis and synthesis of information across multiple classification systems. Can accommodate biological classification as well as other systems (e.g., geology, vegetation)

Metadata for documents, data sets, articles, images; links to full-text documents and files; allows searching all content including PDFs. Retrieve data, documents, other file types.

Eliminates user logins and passwords. System determines who you are and your specific permissions for data sets and subsets (e.g., who can view certain sensitive data, edit or delete particular records, upload data)

Request and receive customized notifications (via email or other means) when specific events occur in an information system (e.g., a subject-specific record added or changed)

Submit suggestions or bugs directly to information system managers; progress of resolution visible. Engages users directly in improving information system and helps ensure it meets their needs.

UnitsInformation on designated geographic or administrative areas such as parks, wilderness, or regions. Includes ability to link and cross-link units (e.g., one unit may comprise many units)

Organization - StructureTeams

• Functional• Architecture• User Interface

Design• Development• Quality Assurance• Infrastructure

• … and Project Management

Project Management

Configuration Management

Functional AnalysisChange Control Board

Quality Assurance Development

Architecture

Iterative Development Process

3-4

we

ek

itera

tion

s

User

User

Person 1

User

User Community

Organization - Methodology

Rational Unified Process• Project plan is divided into iterations

• Project plan tackles high risk in first few iterations

• An iteration is divided into multiple tasks (or work items)

• Each iteration results in an executable application or service

• Each iteration consistently delivers demo-able software

Quality Assurance Development

Architecture

Iterative Development Process

3-4

wee

k ite

ratio

ns

Building the IRMA Infrastructure

Testing Role-based Teams

User Interface Consistency

User Requirements

Iterative Development

Reporting Portal Help AuditingError

Handling

Foundation

Processes

Frameworks

Supporting ServicesIdentity Feedback Notification Park Units File

Resource DataSpecies References Observations Vouchers Taxonomy

Hardware Software Staff Organization

NPS Experience with SOA – Key Points

• Required changes in how people worked together and our development approach;– SOA is something you do, not something you buy– Organized people into functional teams, rather than one

developer per application

• Structural change: Decoupled the “what” from the “how”– “What”: Analysts focus on user requirements– “How”: Developers focus on technology

• End Result:– Improved applications & services; happier users– Lets you quickly build custom business processes from existing

services (e.g. service orchestration) The organization can survive loss of key players

– Is a sophisticated undertaking, but well worth it

Natural Resource Database Template

• NRDT standardizes data elements that are common to most I&M data sets (e.g., locations, events)

• Allows for higher-level integration of data sets across networks or regions (rolling up the data)

• Allows for linkage to GIS• More efficient application development• Users across the country can share code, share

approaches or solutions to specific problems, and use a common database vocabulary.

A standardized yet flexible database core for developing natural resource-related applications

Metadata

CoreTables

Protocol-specific Tables

Shapefile

Optional GISFeatures

Coverage

Geodatabase

Core Tables

NRDT Front-end Application Builder (FAB)

• A Microsoft Access file that provides a user-interface and administrative tools for developing NRDT applications

– is linked to NRDT back-end database file– automates table-linking, compaction, backups, and

lookup table management– provides main menu, data gateway form for selecting

records, and data entry forms for entering new records– works with NRDT back-end files with any of the primary

key options

Natural Resource Program CenterInventory & Monitoring Program

NRDT Front-end Application Builder

•User-interface template for NRDT•Includes many built-in features and frequently-used utilities•Gives applications a standardized look-and-feel

San Francisco Area Network Sonoran Desert Network

• Currently available to NPS users; read-only, non-sensitive data• Additional functionality and data added every few months• Legacy systems no longer needed by summer 2010• Non-sensitive data expected to be publically available in 2010

Simple search tool results in list of references and attached files if available

250,000+ records now available.

Natural Resource Documents and Data (References)

Simple search tool results in certified species list and related data. Users can view, print, download results.

Park Species Information

Taxonomy service available to search and crosswalk with multiple taxonomic systems.

Taxonomic Information

NPS Natural Resource & GIS ProgramsInventory and Monitoring Program

FWS Meeting - Sept 2009

IRMA Integration


FWS Meeting - Sept 2009

NPScape Measures

• Habitat availability• Fragmentation and isolation• Watershed health• Anthropogenic threats to park resources

Land CoverEnhanced NLCD 1992NLCD 1992NLCD 2001NLCD Change ProductNLCD Impervious SurfaceNLCD Tree CanopyHistoric Natural Fire RegimeLandFire - all productsLand cover diversity (Simpson's)NatureServe Ecological SystemsGAP/ReGAPLand cover characteristicsForest fragmentationMorphological pattern metricsForest cover typesConverted and natural landcover

LandformDEM - 10, 30, and 120 meter

Slope and aspect

Depth to bedrock

Sand, silt, clay fractions

Crop capability

Geology

Climate Precipitation Temperature (min/max, variability) Growing season daysNDVISea surface temperature

TransportationRoads (Multiple data sources)Railroads (U.S. and Canada)National Waterways

PopulationNightime lights (1992/93, 2000)Populated placesU.S. citiesU.S. urban areas1990, 2000 Census, block groupPopulation projections by countySEDAC census gridsHousing density (1940-2040)Inventoried roadless areaAgriculture census by countyWater use by countyConservation risk indexWildland Urban Interface

BoundariesOmernick Ecoregions (and CEC)Bailey Ecoregions Physiographic ProvincesUNEP Large Marine Ecosystems StatesCounties NPS Units (with various buffers)NPS Vital Sign NetworksProtected Areas BoundariesFederal LandsNational Wilderness Preservation SystemContinental DivideNCDC Climate DivisionsNEON Domains

HydrologyHydrologic Units (4, 6, 8, 12-digit)NHD (med and high resolution)ImpoundmentsAquifersGround water climate response networkSea ice (North America)

Data Sets that we have available:

Office of Inventory, Monitoring, and EvaluationNatural Resource Program Center


FWS Meeting – Sept 2009

Integration Approaches for Monitoring Data

• NRDT upload and map display functions– Upload and centralize data from desktop databases– Upload monitoring locations and attributes:

• Visualize and query– Access MDB SS2008 ArcSDE/ArcGIS Server– Eventual integration within IRMA frameworks


FWS Meeting – Sept 2009

Upload from NRDT

Run demo

Inventory & Monitoring Program“In just a few short years, the I&M networks have become known as a key source and supplier of reliable, organized, and retrievable information about parks that was formerly unavailable, misplaced, or lost to managers and others who needed the information for sound decisions or sound science.”

FY 2010: Accelerate development Packaging & Delivery of data for modeling Close partnership with USFWS

DOI Climate Change Task Force CC Impacts Data Collection, Integration, and Management Workgroup...

Documents

Transcript of DOI Climate Change Task Force CC Impacts Data Collection, Integration, and Management Workgroup...