Waters Observatory Network

• Observe water systems

• Observe in many different contexts and for different purposes

• Have a Waters Network Information System for sharing data– Common language for data– Geographic federation of dispersed data

sources through web services

Project co-PI in Phase 2

Collaborator in Phase I

CUAHSI HIS Partner Institutions

10 WATERS Testbed Sites

HISTeam

WATERSTestbed

WATERS Network Information System

SuperComputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

MicrosoftCUAHSI HIS

HIS Team and its Cyberinfrastructure Partners

HIS Team:Texas, SDSC,Utah, Drexel,

Duke

SuperComputerCenters:NCSATACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

Microsoft

HISTeam

WATERSTestbed

WATERS Network Information System

CUAHSI HIS

HIS and WATERS

CUAHSI Member Institutions

116 Universities as of November 2006

SupercomputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

OpenMI

HISTeam

WATERSTestbed

WATERS Network Information System

CUAHSI HIS

HIS, WATERS and the CUAHSI Community

KISTERS Water Resources Management Data System

KISTERS is building CUAHSI web services onto their database

Stefan Fuest

SupercomputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

OpenMI

HISTeam

WATERS Testbed

WATERS Network Information System

CUAHSI HIS

International Partners

CSIRO Land and Water ResourcesWater Resources Observations Network

(WRON)

European CommissionWater database design and model integration

(HarmonIT and OpenMI)

HIS Goals

• Hydrologic Data Access System – better access to a large volume of high quality hydrologic data

• Support for Observatories – synthesizing hydrologic data for a region

• Advancement of Hydrologic Science – data modeling and advanced analysis

• Hydrologic Education – better data in the classroom, basin-focused teaching

Rainfall & SnowWater quantity

and quality

Remote sensing

Water Data

Modeling Meteorology

Soil water

Water Data Web Sites

NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454

Time series of streamflow at a gaging station

USGS has committedto supporting CUAHSI’sGetValues function

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

Water Quality Measurement Sites in EPA Storet

Substantial variation in data availability from states

Data from Bora Beran, Drexel University

Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

Geographic Integration of Storet and TCEQ Data in HIS

Observations CatalogSpecifies what variables are measured at each site, over what time interval,

and how many observations of each variable are available

Point Observations Information Model

Data Source

Network

Sites

Variables

Values

{Value, Time, Qualifier}

USGS

Streamflow gages

Neuse River near Clayton, NC

Discharge, stage (Daily or instantaneous)

206 cfs, 13 August 2006

• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value

Data Discovery and Delivery

Data Source

Network

Sites

Variables

Values

Observations metadata

Observations data

HIS ServerObservations

Catalog

Web services

Data Discovery

Data Delivery

• HIS facilitates data discovery by building and maintaining observations catalogs• Data delivery occurs through web services from remote data archives or local observations databases. Water resource agencies support data delivery services.

NWISNWIS

ArcGISArcGIS

ExcelExcel

NCARNCAR

UnidataUnidata

NASANASAStoretStoret

NCDCNCDC

AmerifluxAmeriflux

MatlabMatlab

AccessAccess JavaJava

FortranFortran

Visual BasicVisual Basic

C/C++C/C++

Some operational services

CUAHSI Web ServicesCUAHSI Web Services

Data SourcesData Sources

ApplicationsApplications

Extract

Transform

Load

http://www.cuahsi.org/his/

WaterOneFlow Web Services

Data access through web

services

Data storage through web

services

Dow

nlo

ads

Upl

oa

ds

Observatory servers

Workgroup HIS

SDSC HIS servers

3rd party servers

e.g. USGS, NCDC

GIS

Matlab

IDL

Splus, R

D2K, I2K

Programming (Fortran, C, VB)

Web services interface

Web portal Interface (HDAS)

Information input, display, query and output services

Preliminary data exploration and discovery. See what is available and perform exploratory analyses

HTML -XML WS

DL

- SO

AP

Hydrologic Information System Service Oriented Architecture

NWISNWIS

ArcGISArcGIS

ExcelExcel

NCARNCAR

UnidataUnidata

NASANASAStoretStoret

NCDCNCDC

AmerifluxAmeriflux

MatlabMatlabAccessAccess SASSAS

FortranFortran

Visual BasicVisual Basic

C/C++C/C++

CUAHSI Web ServicesCUAHSI Web ServicesIlya Zaslavsky

My server

Web services: key terms

Web ServiceCode

WSDL Web service description language Your Web application

Reference a web service, and call a WS method

Your desktop application

Reference a web service, and call a WS method

SOAP Messages

SOAP: Simple Object Access Protocol

- From different database structures, data collection procedures, quality control, access mechanisms to uniform signatures … Water Markup Language- Tested in different environments- Standards-based- Can support advanced interfaces via harvested catalogs- Accessible to community- Templates for development of new services- Optimized, error handling, memory management, versioning, run from fast serversAnd: working with agencies on setting up services!

Web service testing pages

http://water.sdsc.edu/webservicetestpage/

For Developers and Information Managers

• Code developers– HIS Team System: code repository, web services– Design Documents– Workbook for using web services in different environments– Web service testing and versioning guidelines and modules– Support

• Site managers– Step-by-step instructions for setting up the server (eventually, will be an

“HIS appliance”)– Configuration and data management tools

What it takes to setup and maintain HIS server, and how to get there from your current setup – THIS AFTERNOON

CUAHSI Hydrologic Information System Levels

National HIS – San Diego Supercomputer Center

Workgroup HIS – research center or academic department

Personal HIS – an individual hydrologic scientist

HIS Server

HIS Analyst

Map interface, observations catalogs and web services for national data sources

Map interface, observations catalogs and web services for regional data sources; observations databases and web services for individual investigator data

Application templates and HydroObjects for direct ingestion of data into analysis environments: Excel, ArcGIS, Matlab, programming languages; MyDB for storage of analysis data

Data Ingestion

MatlabExcelArcGIS Java .Net

HydroObjects

HIS Analyst

HIS Server

WaterOneFlow Services

Observations Data

Tim Whiteaker

Tutorials and Templates that show how to use Web services in various applications

ftp://ftp.crwr.utexas.edu/pub/outgoing/CUAHSI/HIS_workbook/20061115/

IncludesHydroObjects

Library

HIS Analyst Demo (Tim Whiteaker)

• Ingestion of data using web services into Excel and VB.Net

CUAHSI Hydrologic Data Access System

A common data window for accessing, viewing and downloading hydrologic information

USGSUSGS

NASANASANCDCNCDCEPAEPA NWSNWS

Observatory DataObservatory Data

http://river.sdsc.edu/HDAS

HIS Server

• Supports data discovery, delivery and publication– Data discovery – how do I

find the data I want?• Map interface and

observations catalogs

– Data delivery – how do I acquire the data I want?

• Use web services or retrieve from local database

– Data Publication – how do I publish my observation data?

• Use Observations Data Model

HIS Server Architecture

• Map front end – ArcGIS Server 9.2 (being programmed by ESRI Water Resources for CUAHSI – led by Dean Djokic, an IHE graduate)

• Relational database – SQL/Server 2005 or Express

• Web services library – VB.Net programs accessed as a Web Service Description Language (WSDL)

HIS Server Demo (Dean Djokic)

ESRI Water Resources Applications Group, Redlands CA

(a voluntary contribution to CUAHSI)

Digital WatershedHow can hydrologists integrate observed and

modeled data from various sources into a single description of the environment?

Digital WatershedHydrologic Observation

Data

GeospatialData

Weather and ClimateData

Remote SensingData

(NetCDF)

(GIS)(Relational database)

(EOS-HDF)

Digital Watershed

A digital watershed is a synthesis of hydrologic observation data, geospatial data, remote sensing data and weather

and climate data into a connected database for a hydrologic region

HIS ServersHydrologic

ObservationsServer

GIS Data Server

Weather and ClimateServer

Remote SensingServer

Digital Watershed

HIS Servers provide hydrologic observations, weather and climate, GIS and remote sensing data. For HIS version 1.0, the focus is a hydrologic observations server for data from gages and monitoring sites at point locations.

Water Resource Regions and HUC’s

NHDPlus for Region 17E

NHDPlus Reach Catchments ~ 3km2

About 1000 reach catchments in each 8-digit HUC

Average reach length = 2km 2.3 million reaches for continental US

National Land Cover Dataset

http://landcover.usgs.gov/nationallandcover.html

http://seamless.usgs.gov/Get the data:

Reach Attributes

• Slope• Elevation• Mean annual flow

– Corresponding velocity

• Drainage area• % of upstream

drainage area in different land uses

• Stream order

http://www.ncgc.nrcs.usda.gov/products/datasets/statsgo/

1:250,000 Scale Soil Information

Hydrologic Landscape Regions

USGS Characterization of nature of soil and groundwater systems

http://water.usgs.gov/lookup/getspatial?hlrus

Arc Hydro: GIS for Water Resources

• Arc Hydro– An ArcGIS data model

for water resources– Arc Hydro toolset for

implementation– Framework for linking

hydrologic simulation models

The Arc Hydro data model andapplication tools are in the publicdomain

Published in 2002

Arc Hydro Links Space and TimeIntegrate geospatial data for

land and water systems

You store this informationin a relational database orget it from web services

Link space and timewith HydroID

All Hydrofeatures of a testbed area will have a unique HydroID

Land base from NHDPlus + added data layers

Open Hydro adopts the Arc Hydro data model and exposes it using open source GIS and database tools. Open Hydro also includes a suite of tools for working with hydrologic data and managing the associated hydro geodatabase.

Dan Ames, Idaho State University

• Based on MapWindow GIS: Free open source GIS alternative for data visualization and analysis.

• MapWindow GIS follows OGC standards and works with ESRI data formats.

• MapWindow GIS is extensible using VB.NET, VB6, C#, and ActiveX.

• Sponsored by NASA, NOAA, DOE, and EPA (new platform for BASINS).

• Nearly 30,000 downloaders so far at: www.MapWindow.org

Open Hydro is still under development and ISU is seeking partners to work together on various functions.

Current completed functions include geodatabase import/export, and automatic watershed delineation and support for ESRI’s geography network and online data sources.

http://www.daymet.org/

Demo of NHDPlus and Weather Downloader for the San Marcos

Basin

By Ernest To

Center for Research in Water Resources

• Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them

Objective

NWIS

NARR

NAWQANAM-12

request

request

request

request

request

requestrequest

request

request

return

return

return

return

return

returnreturn

return

return

What we are doing now …..

Michael PiaseckiDrexel University

Semantic MediatorWhat we would like to do …..

NWIS

NAWQA

NARR

generic

request

GetValues

GetValues

GetValues

GetValues

GetValues

GetValuesGetValues

GetValues

GetValues HODM

Michael PiaseckiDrexel University

Demo on Semantic Mediator

By Bora Beran and Michael Piasecki, Duke University

• Project sponsored by the European Commission to promote integration of water models within the Water Framework Directive

• Software standards for model linking• Uses model core as an “engine”• http://www.openMI.org

OpenMI – Links Data and Simulation Models

CUAHSI Observations Data Model as an OpenMI component

Simple River Model

Trigger (identifies what value should be calculated)

Typical model architectureApplication

User interface + engineEngine

Simulates a process – flow in a channelAccepts inputProvides output

ModelAn engine set up to represent a particular location e.g. a reach of the Thames

Engine

Output data

Input data

Model application

Run

Write

Write

Read

User interface

Accepts Provides

Rainfall

(mm)

Runoff

(m3/s)

Temperature

(Deg C)

Evaporation

(mm)

Accepts Provides

Upstream Inflow

(m3/s)

Outflow

(m3/s)

Lateral inflow

(m3/s)

Abstractions

(m3/s)

Discharges

(m3/s)

River Model

Linking modelled quantities

Rainfall Runoff Model

Data transfer at run time

Rainfall runoff

Output data

Input data

User interface

River

Output data

Input data

User interface

GetValues(..)

Models for the processes

River(InfoWorks RS)

Rainfall(database)

Sewer(Mouse)

RR(Sobek-Rainfall

-Runoff)

Data exchange3 Rainfall.GetValues

River(InfoWorks-RS)

Rainfall(database)

Sewer(Mouse)

2 RR.GetValues

7 RR.GetValues

RR(Sobek-Rainfall

-Runoff)

1 Trigger.GetValues

6 Sewer.GetValues

call

data

4

5 8

9

OpenMIOpenMI = Open Modeling Interface

It is a Standard for Model Linking

It is part of HarmonIT, a European Commission funded research project with 14 project partners (universities & software companies)

www.openmi.org www.harmonit.org

OpenMI Conceptual Framework

VALUES

All values are referenced in a what-where-when framework, allowing different data resources or models to communicate data

HIS as OpenMI Components

water balance model

NWISStreamflow

DaymetPrecipitation

To calculate storage, the model needs inflow and outflow

To calculate storage, the model needs precipitation

Goal: Link the National HIS web services with a simple water balance model using OpenMI as the mediator

Trigger:

Calculate storage

Demo on Intregating Web Services and OpenMI to make a

water balance model

By Jon Goodall, Duke University

Watershed Hydrovolumes

Geovolume is theportion of a hydrovolumethat contains solidearth materials

USGS Gaging stations

Hydrovolume

Stream channel Hydrovolumes

Need the capacity to represent Acoustic Doppler Current Profiler (ADCP) data (Iowa)

Residence time distributions(Minnesota)

Integration of surface water and groundwater data

• Describe the relationship between surface water features ( e.g. streams

and waterbodies) with groundwater features (aquifers, wells).

• Enable the connection with the surface water data model

Hydro network Aquifers

In the future go to 3D...

Water OneFlow• Like Geospatial OneStop, we need a “Water

OneFlow” – a common window for water data and models

• Advancement of water science is critically dependent on integration of water information

Federal

AcademicLocal

State