1

Space-Time Datasets in Arc Hydro II

by Steve Grise (ESRI), David Maidment, Ernest To, Clark Siler

(CRWR)

2

CUAHSI Observations Data Model

Space-Time Datasets

Sensor and laboratory databases

From Robert Vertessy, CSIRO, Australia

3

Space-Time Dataset

• A set of records with – Time – Location– 1 or more variables

time

variables

x, y,

z

x, y,

z

x, y,

z

cba

cba

cba

x, y,

z

cba

x, y,

z

cba

x, y,

z

cba

4

Example: River Flow• For surface water resources, stream gages have a fixed location with

continuous measurements over time• Variables related to stream flow are the most common measurements• Data is typically measured regularly and continuously, but there are often

gaps due to device errors or routine maintenance• There are also cases of overflow or dry conditions where the values are

outside of the range of measurement for the device

time

variables

fixed

x, y,

z

cba

cba

cba

cba

cba

cba

cba

cba

cba

cba

cba

Data gap

cba

cba

An overflow condition could be recorded simply as > 500 cubic feet/second

stream flowriver heightmean velocity

5

Example: Water Quality• For water quality, sampling sites have a fixed location with intermittent measurements

over time– Four times per year is typical

• There is a sampling “event”, and a large number of chemical species are produced through laboratory analysis of water samples

• Data has metadata that specifies what laboratory procedure was used• Some data require a qualifier to be properly interpreted like “<“ to indicate a

measurement that is below a detection limit• Data are “Time stamped” with the time that the sampling event began. They are

considered “instantaneous data” observed at that time.

time

variables

fixed

x, y,

z

cba

cba

cba

cba turbidity

nitrateconductivityc

ba

t1 t2 t3 t4 t5

water quality sample

6

Display of data that vary in latitude, longitude, depth and time (Ernest To)

7

Data Structure for a single variable

These data are extracted from CUAHSIODM, and Offset = Depthin this instance

8

Example: Water Reservoir• For water reservoirs, data is recorded for the water level of the reservoir, along with

all inflows and outflows• A flow time series dataset describes the information required to do a water balance

on the reservoir contents• “Flow variables” apply over the entire time interval; “state variables” apply at instants

of time at beginning and end of interval;• Typically there are derived datasets

– Monthly data compiled from daily data– Annual data from monthly data

• Data are recorded regularly through time

time

variables

x, y,

z

cba

cba

cba

cba

cba

cba

cba

cba

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

inflowoutflowstorage

Inflow

Outflow

Precip

Evap

Storage

9

Example: Water Rights Analysis• A water resources simulation model is run for monthly time steps for ~50 years and it

computes ~40 variables related to water supply reliability– Water rights diversion points, – Reservoirs, and – Other “control points” on the stream system

• Each model “run” generates millions of data values.• The “data cube” is completely filled in because it is all computed• Information products needed are graphs of variables at points, maps of feature

conditions at a single time point, and maps of averages through a defined time interval of feature conditions (i.e. dataset derived “on the fly”)

time

variables

x, y,

z

cba

cba

cba

cba

cba

cba

cba

cba

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

% of time reliability% of volume reliabilityflow

Study area (watershed)

modeled point features

10

Maps and Charts

Plot a map for a time point Plot a graph for a space point

Space Time A set of variables ……

11

Example: Climate and Weather• Observations that come from weather balloons and other measuring

devices have dynamic location properties • For weather and climate forecast datasets, each data point represents an

area with consistent atmospheric characteristics• For weather observations, a large amount of data comes from fixed stations

so the datasets are similar to stream gage datasets

time

variables

x, y,

z

cba

cba

cba

cba

cba

cba

cba

cba

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

temperatureair pressurerelative humidity

tttt

t

tt

t

balloon trajectory

forecast data

12

Example: Species Observations• In this type of dataset, observers are frequently moving

along a path such as a hiking trail or a boat cruise • Multiple species may be observed, and even the lack of

information is significant• Data is often recorded using offsets from the observer

location

time

variables

species group “a”species group “b”species group “c”

a a

c

a

c cb

aa

acc

c

b x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

x, y,

z

13

Other Datasets• There are many types of Time Series Datasets

– Observations– Samples– Model results– Remote sensing data/imaging

• Concepts are useful for many communities– Science– Business– Statistics– Planning– Health– Transportation

14

Space-Time Datasets:Implementation Concepts

• The general pattern can be described as– Time Series Values

• The data

– Time Series Descriptions • The metadata

• There are a number of ways to store and manage this information in a computer system

*

1

Time StepTime UnitIs RegularData TypeData OriginTime Reference SystemLocation Reference System

Time Series Description

1 *

VariableVariable UnitsVariable Unit Type

Variable Description

DateTimeLocationVariable[n]

Time Series Values

1

*

15

Example: Arc Hydro Version 1 Implementation

• Approach works well for an individual project with stream gage and other surface water data

• Constrained to 1 variable per time step• Limited in its ability to handle location

– Changes in x, y, z over time• i.e., Marine and species observation

datasets have an additional “cruise” or “observation” concepts linking multiple features

• FeatureID provided some flexibility, but did not directly support unique identity for features at different time steps

• In general, implementation patterns for the feature portion of the data model were not explored/explained

– Different spatial representations• Raster data• Multidimensional data

– GIS Layers and their properties were considered but not explained

– Inefficient approach with multiple variables

*

1

FeatureIDTSTypeIDTSDateTimeTSValue

Time Series TableTimeSeries

TSTypeIDVariableVarUnitsUnitTypeIsRegularTimeStepTimeUnitDataTypeOrigin

Time Series Type TableTSType

16

Arc Hydro Version 2Improvements

1. GIS Layer and representation focus

2. Use of Metadata

3. Improved Efficiency

4. More documented implementation patterns

5. General Time Series Dataset concepts applicable to many communities

17

Representations in GIS

• Time series data can be represented in different ways– Charts and graphs– Modeling simulations– Surfaces– Rasters– Vector feature classes

• GIS Layers provide a convenient set of representation types for different views into Time Series Datasets

18

Layers

• Layers represent data– Layer Properties

• Queries• Representation types• Display/styles• Variable(s)• Labels

• Layers deal with presentation of data, and they are closely linked to the data storage model

19

Vector Layers

Feature Series

A Feature Series is a collection of features indexed by time. Each feature in a feature series exists for only a period of time, making Feature Series an ideal structure for representing a series of flood inundation polygons. Feature Series can also be used to represent the movement of particles through the environment. In this case, the Feature Series would be a set of points, each valid for some instant in time.

FeatureIDFeatureGroupIDX, Y, ZTimeVariable

Multipatch Layer

FeatureIDFeatureGroupIDX, Y, ZTimeVariable

Polygon Layer

FeatureIDFeatureGroupIDX, Y, ZTimeVariable

Line Layer

FeatureIDGroupIDX, Y, ZTimeVariable

Point LayerTime LocationVariable[n]

Time Series Values

Derived*

20

Raster Layers

Raster Series

Stored* 1

Raster Series are collections of rasters indexed by time. Each raster is a "snapshot" of the environment at some instant in time. Grouping a series of rasters can describe how the environment changes over time. Raster Series are useful for describing the dynamics of spatially continuous phenomena, like ponded depth in the Everglades, or rainfall measured by NEXRAD.

Raster Catalog

RasterNameVariableTime

VariableXDimensionYDimensionOutputNameDimensionZvariableMVariable

Raster Layer

TimeLocationVariable[n]

Time Series Values

TimeLocationVariable[n]

Multi-Dimensional Dataset

Derived *

VariableTime

Raster Dataset

Stored*

1

21

Metadata• Each Time Series Dataset is a

complex structure, and there are many patterns

• Metadata is a tool that can be used to document datasets

– Facilitates search and discovery– Aids in sharing and re-use of data – Standards-based

metadata/cataloging methods are available

• In practice, once users understand the dataset, they tend to work with the Time Series Values and rarely re-visit the metadata in applications

• Shift in Arc Hydro II to use of FGDC/ISO metadata to document datasets and variables

– For the grey boxes in the diagram shown here

*

1

Time StepTime UnitIs RegularData TypeData OriginTime Reference SystemLocation Reference System

Time Series Description

1 *

VariableVariable UnitsVariable Unit Type

Variable Description

DateTimeLocationVariable[n]

Time Series Values

1

*

22

Improved Efficiency• In Arc Hydro 1, we tried to put all time series values into a single

table• This implied creating rows for each variable, or adding additional

columns/TSValues rows to datasets• Since it was table-based, it did not include feature and raster

representations, which required additional processing steps• By promoting multiple datasets with a flexible approach for

managing variables, data management activities will be improved, especially for larger datasets

*

1

FeatureIDTSTypeIDTSDateTimeTSValue

Time Series TableTimeSeries

TSTypeIDVariableVarUnitsUnitTypeIsRegularTimeStepTimeUnitDataTypeOrigin

Time Series Type TableTSType

PolygonPoint Line

RasterCatalog

RasterDataset

Table

Multi-Dimensional

Dataset

Multi-Patch

Single Time Series Table with 1 variable Time Series Datasets with multiple variables

23

Improved Efficiency

• For display, layers are built using Time Series Datasets

• Typically we “Select” or “Slice” 1 variable for presentation

• Layers can be built from source Values using InMemory layers, or built from Time Series Datasets

Time Series Layers with variable(s)

Time Series Datasets with variable(s)

VariableXDimensionYDimensionOutputNameDimensionZvariableMVariable

Raster Layer

FeatureIDFeatureGroupIDX, Y, ZTimeVariable

Multipatch Layer

FeatureIDFeatureGroupIDX, Y, ZTimeVariable

Polygon Layer

FeatureIDFeatureGroupIDX, Y, ZTimeVariable

Line Layer

FeatureIDGroupIDX, Y, ZTimeVariable

Point Layer

Time LocationVariable[n]

Time Series Datasets

*Time LocationVariable[n]

Time Series Values

Derived

24

Implementation Patterns

• Patterns will be explained for different types of implementations– Small/single project– Workgroup or multi-project environments– Very large datasets– Different spatial representation options– …– One key difference is that there will be multiple

datasets – basically one dataset per set of time series values

• Different dataset names and storage strategies• Documented with metadata

25

A General Spatial-Temporal Model

• A Space-Time Dataset is a set of records with – Time – Location– 1 or more variables

time

variables

x, y,

z

x, y,

z

x, y,

z

cba

cba

cba

x, y,

z

cba

x, y,

z

cba

x, y,

z

cba