November 2010Deltares Configuration Course1 Day 2.
-
Upload
rosalind-mosley -
Category
Documents
-
view
218 -
download
1
Transcript of November 2010Deltares Configuration Course1 Day 2.
November 2010Deltares Configuration Course 1
Day 2
November 2010Deltares Configuration Course 2
TimeSeriesSets
November 2010Deltares Configuration Course 3
FEWS Time Series
Time series are available from two source:
external
simulated
Time series are in two categories in relation to time:
historical (continuous in time)
forecasting (characterised by its start time)
Time series can be in four formats:
0D – scalar
1D – vector or longitudinalprofile
2D – grid
2D – polygon
Time Series are handled in the form of Time Series Sets
November 2010Deltares Configuration Course 4
FEWS Time Series – external historical
00:00
04:00
07:00
10:00
Complete series
. Added incrementally
. Can be edited by the user
. Only new and changed values are stored
Complete series
Manual edit
Time of import
November 2010Deltares Configuration Course 5
FEWS Time Series – external forecasting
Time of Forecast
. Added and stored individually
. Can be edited by the user
. Usage of series depends on T0
00:00
04:00
08:00
Series at 08:00
Series at 06:00
00:00 04:00 08:0006:00
November 2010Deltares Configuration Course 6
FEWS Time Series – simulated historical
System Time
. Continuous in time
. Referenced by the forecast model that produced them
. Approved series shown automatically (including history)
. Most often linked by model states
04:00
08:00
12:00
Series at 16:00
Series at 08:00
00:00 04:00 08:00 12:00
16:00
16:00
November 2010Deltares Configuration Course 7
FEWS Time Series – simulated forecasting
System Time
. Continuous in time (in combination with simulated historical)
. Referenced by the forecast model that produced them
. Approved and selected series can be viewed
04:00
08:00
12:00
Series at 12:00
00:00 04:00 08:00 12:00 16:00
November 2010Deltares Configuration Course 8
FEWS Time Series – temporary
temporary time series
time series used in pre- / post-processing not stored in database
November 2010Deltares Configuration Course 9
Time Series Sets
Properties of time series:
• location Id – or – locationSetId• parameter Id• timestep• relativeViewPeriod
• valueType (scalar, grid, profile, ..)• timeSeriesType (external historical, external forecast, ..)• moduleInstanceId (import, PDM, KW, ISIS, ..)• readWriteMode
• expiryTime (when to delete from database)• synchLevel
• simple transformations (delay, multiplier, divider, incrementer)• ensembleId
November 2010Deltares Configuration Course 10
Time Series Sets
• All data handling is done through Time Series Sets
• Some elements are only used when retreiving time series, other elements only when storing data
November 2010Deltares Configuration Course 11
Time Series Sets - Examples
Import Run General Adapter Run
November 2010Deltares Configuration Course 12
FEWS Workflows and Modules
November 2010Deltares Configuration Course 13
FEWS Workflows
• Workflows are logical sequences of running forecast modules• Workflows must be registered in the workflow descriptors
– Module Instance ID– Sub-workflows– Run independent– Fallback activity– Ensemble
November 2010Deltares Configuration Course 14
FEWS Modules and Module Instances
. Modules
are FEWS plug-ins that can be used in a workflow are registered in moduleDescriptor file as Java Classes are part of the system configuration
. Module Instances
are configured modules of a particular module are registered in moduleInstanceDescriptors file are part of the regional configuration
November 2010Deltares Configuration Course 15
Sub workflows
Workflows can be nested:• Workflow “DO_ALL”
– (sub)Workflow “DO_Step1”– (sub)Workflow “DO_Step2”
Where:• (sub)Workflow “DO_Step1”
– ModuleInstance “DO_Step1a”– ModuleInstance “DO_Step1b”– ModuleInstance “DO_Step1c”
• (sub)Workflow “DO_Step2”– ModuleInstance “DO_Step2a”– ModuleInstance “DO_Step2b”– ModuleInstance “DO_Step2c”
FEWS Explorer:
. Manual Forecast Display Workflow “DO_ALL” (sub)Workflow “DO_Step1” (sub)Workflow “DO_Step2”
November 2010Deltares Configuration Course 16
Introduction to some available modules
. Import
. Export
. Transformation
. General Adapter
. Lookup
. ErrorModel
. Correlation
. PerformanceIndicator
. ThresholdEvent
. Report
November 2010Deltares Configuration Course 17
FEWS Workflows
FEWS Workflows can be run (and debugged) on module basis (via F12)
November 2010Deltares Configuration Course 18
Importing data
November 2010Deltares Configuration Course 19
Flood Forecasting processing steps
Create evaporation profiles for hydrological model
Calculate catchment average precipitation based on rain
gauge data
Import data from external sources
Combine precipitation sources (e.g. Rain gauge, Radar data,
Meteorological Forecast models)
Run hydrological model to calculate discharge for
catchments
Run hydrodynamic model to calculate water level for
forecast points
Prepare input for hydrodynamic model based on hydrological
models, gauging stations, tidal boundaries
Run error correction model to update output form
hydrodynamic model
November 2010Deltares Configuration Course 20
Importing data
Import Modules allows data from external sources to be imported
. Various Scalars – e.g. PI XML and Wiski
. ASCII grid time series
. Binary GRIB (grid) and BIL time series
. Currently around 50 data import types
FEWS imports files from a directory specified in the global.properties
November 2010Deltares Configuration Course 21
Import Module
Key Features:
. ID mapping
. Flag mapping
. Units mapping
. Time zone mapping
. Tolerance
. Validate
November 2010Deltares Configuration Course 22
EA Import Module
November 2010Deltares Configuration Course 23
Import Module – ID mapping
• Matching ‘outside world’ to the ‘FEWS world’– locations (ids)– parameter (ids)
• Three levels– direct mapping location names AND parameter names differ– indirect either location names or parameter names differ– 1:1 location and parameter names do NOT differ
• Used when importing and exporting time series
November 2010Deltares Configuration Course 24
Id Mapping
• Direct
• Indirect
• 1:1
November 2010Deltares Configuration Course 25
Import Module – Unit conversion
• Map internal units to external units• Used when importing and exporting time series• Extra option to convert datum of level recordings (external unit mAOD)
November 2010Deltares Configuration Course 26
Import Module – Flag conversion
• Map internal quality flags to external quality flags• Used when importing and exporting time series
November 2010Deltares Configuration Course 27
Import Module – other elements
Time zone mapping
database stores all data in GMT, specify timezone when importing
Tolerance
when imported data is not exactly observed at the cardinal time step
Validate
Imported files can be validated against their own schema
November 2010Deltares Configuration Course 28
Import Module - Exercise
Import Time series from EA-XML format (specific Environment Agency format)
• Analyse the 5 time series– 2 * Observed_waterlevel for Hydro1 and Hydro2– 2 * Rainfall for Raingauge and Meteostation– 1 * Temperature for Meteostation
• Add elements to ID mapping table• Run import workflow• Analyse data with the Time Series Display
November 2010Deltares Configuration Course 29
Validating Data
• Allow quality checking of all scalar time series data
• Difference in hard limit and soft limit checks– Hard limit checks mark data as unreliable– Soft limit checks mark data as doubtful
November 2010Deltares Configuration Course 30
Validating Data
• Option to enter monthly limits• Rule can be specified for one time series or for a complete set
November 2010Deltares Configuration Course 31
Validating Data - Exercise
• Check the data just imported• Define validation rules• Enter the rules in the ValidationRuleSets XML file• Delete the data store• Run import workflow• View data• Check the difference!
November 2010Deltares Configuration Course 32
Thresholds
November 2010Deltares Configuration Course 33
Example (Map)
Willington Cableway (Midlands, UK)
!
T0
November 2010Deltares Configuration Course 34
Example (Graph)
Willington Cableway (Midlands, UK)
November 2010Deltares Configuration Course 35
Thresholds Utility
The thresholds utility checks when time series cross thresholds
Observed and simulated time series Icons are shown on the map Graphs/lines cross threshold lines Events can be triggered (MC) Up events and down events
Three configuration files
thresholdWarningLevels.xml definition of (unique) thresholds + details thresholds.xml the grouping of the (selected) thresholds thresholdValueSets.xml timeseries (loc/par.) and actual levels info
November 2010Deltares Configuration Course 36
Thresholds - thresholdWarningLevels
• Unique threshold information– colors: displays, reports– icons
• normal/warning• historic colour• forecast colour !
November 2010Deltares Configuration Course 37
Thresholds Utility - Thresholds
• Thresholds usage– groups– selection(s)
November 2010Deltares Configuration Course 38
Thresholds Utility – Thresholds Values Sets
• Detailed threshold information– per location/parameter– actual values to be crossed for the different thresholds– ‘triggers’ (action events for up- or downcrossings)
November 2010Deltares Configuration Course 39
Thresholds – Exercise
Add thresholds for the hydro locations
November 2010Deltares Configuration Course 40
Presentation of data in FEWS
November 2010Deltares Configuration Course 41
Display of data - Filters
November 2010Deltares Configuration Course 42
Display of data - Filters
Filters are used to:– define the locations that are displayed on the map– define the locations that are displayed in the list box– define the parameters that are displayed in the list box
November 2010Deltares Configuration Course 43
Display of data - Filters
Filters • organised in folders with child filters• arrange time series sets in logical groups• import elements in the time series set
– extra options• view permissions• validation icons visible• link to map extent
November 2010Deltares Configuration Course 44
Display of data - Filters
Allow/Disallow editing
Relative View Period for Icon
on main map
synchLevel – for live system
(5=edited data)
November 2010Deltares Configuration Course 45
Display of data - Filters
child filter organised in sub-folders (childs)
November 2010Deltares Configuration Course 46
Time Series Display
Time Series Display is used to show data in a graph and in a table
• Editing and viewing data• Pre-configured displays• Thresholds• Historic Events
November 2010Deltares Configuration Course 47
Time Series Display
Time Series Display uses 4 configuration files
– TimeSeriesDisplayConfig– Thresholds and ThresholdValueSets– DisplayGroups– HistoricalEvents
November 2010Deltares Configuration Course 48
Time Series Display - TimeSeriesDisplayConfig
• default view period• time markers• parameters• module instance mappings
November 2010Deltares Configuration Course 49
Time Series Display - DisplayGroups
Display groups• are pre-defined displays• are used to create folder structure of displays• can be nested
November 2010Deltares Configuration Course 50
Time Series Display - DisplayGroups
display element contains time series to display
multiple sub plots
for each subplot – time series
Options– minimum– maximum– convertDatum– relativeViewPeriod– nrOfRecentForecasts
November 2010Deltares Configuration Course 51
Time Series Display - Exercise
Exercise: Add pre-configured displays
• Add a new display group for the new locations• Add a new display for the hydro locations• Add a new display for the meteo locations
November 2010Deltares Configuration Course 52
FEWS Data manipulation modules
November 2010Deltares Configuration Course 53
Flood Forecasting processing steps
Create evaporation profiles for hydrological model
Calculate catchment average precipitation based on rain
gauge data
Import data from external sources
Combine precipitation sources (e.g. Rain gauge, Radar data,
Meteorological Forecast models)
Run hydrological model to calculate discharge for
catchments
Run hydrodynamic model to calculate water level for
forecast points
Prepare input for hydrodynamic model based on hydrological
models, gauging stations, tidal boundaries
Run error correction model to update output form
hydrodynamic model
November 2010Deltares Configuration Course 54
Interpolation/Transformation Module
Interpolation Module was separate, but now integrated with Transformation Module
The Interpolation module generates data at desired locations or at desired points in time by means of
Serial Interpolation methods
linear interpolation
block interpolation
default value
extrapolation
Spatial Interpolation methods
Kriging
Inverse Distance
bi-linear interpolation
grid cell averaging over catchments
November 2010Deltares Configuration Course 55
Interpolation Module – serial interpolation
November 2010Deltares Configuration Course 56
Interpolation Module – Exercise
• Analyse module that fills gaps in the water level time series• Add the new module instance to the module instance descriptors• Configure a workflow that executes the interpolation module instance• Check results!
November 2010Deltares Configuration Course 57
Interpolation
Possible problems / issues of note
Time Series Set to Interpolate
ModuleInstanceId of series!!! (ImportTelemetry)
Relative View Period
When interpolating to fill gaps
Input time series same as output time series
Using LocationSets and/or Locations
Linear interpolation – can be done using a locationSet
Default – take care as default value may vary per location
November 2010Deltares Configuration Course 58
Transformation Module
. Concepts of transformations function
. User defined function
. Rating curve
. Catchment averaging
. Data Hierarchy
. Typical Profiles
. Aggregation / dis-aggregation
. Equidistant to non-equidistant
. Exercise: Adding rating curves
November 2010Deltares Configuration Course 59
Transformation - Configuration
November 2010Deltares Configuration Course 60
Transformation – Input/Output
Inputs and Outputs are defined as TimeSeriesSet4
November 2010Deltares Configuration Course 61
TransformationUser defined functions and rating curves
Different function / parameters per segment / range
Define ranges in rating curves file / coefficientsets file
November 2010Deltares Configuration Course 62
Transformation – User defined functionSchema
November 2010Deltares Configuration Course 63
Transformation – User defined functionExample
November 2010Deltares Configuration Course 64
Transformation – User defined functionFunctions
Generic function parser used
“Any” function can be defined
November 2010Deltares Configuration Course 65
Transformation – Rating curve
. Same approach as other transformations using stageDischarge class
. Convention on inputId / ouputId & parameter names
. Rating curves are contained in the CoefficientSetFiles (RatingCurves.xml)
November 2010Deltares Configuration Course 66
Transformation – stageDischarge (Rating curve)
Rating curves are linked to locationsOne location orlocationSet
Use coefficients from CoefficientSetFile
November 2010Deltares Configuration Course 67
Transformation – stageDischarge (Rating curve)
• StageDischargeCoefficientSets.xml (name not fixed) • In CoefficientSetsFiles folder• Referenced in Transformation ModuleInstance• Rating curve defined only once
Referenced by locationId
November 2010Deltares Configuration Course 68
Transformation – Weighted averaging
Convention on inputId / ouputId & parameter names formula: (X1 * a + X2 * b + X3 * c) / (a + b + c) number of locations is flexible, missing locations will be left out
November 2010Deltares Configuration Course 69
Data Hierarchy
Fallback options to complete a series
used first
used second
used third
November 2010Deltares Configuration Course 70
Transformation – Data Hierarchy / Merge
used first
used second
result
November 2010Deltares Configuration Course 71
Transformation – Typical profile
Examples of typical profile use:
Evaporation series (one value per calendar day) Temperature variation across day
November 2010Deltares Configuration Course 72
Transformation – Typical profileFixed data/time
November 2010Deltares Configuration Course 73
Transformation Module - Exercise
Add rating curve for location Hydro1
Analyse transformation file
register module instance in descriptors file add module instance to the exercise workflow
November 2010Deltares Configuration Course 74
FEWS External Modules/Models
November 2010Deltares Configuration Course 75
Flood Forecasting processing steps
Create evaporation profiles for hydrological model
Calculate catchment average precipitation based on rain
gauge data
Import data from external sources
Combine precipitation sources (e.g. Rain gauge, Radar data,
Meteorological Forecast models)
Run hydrological model to calculate discharge for
catchments
Run hydrodynamic model to calculate water level for
forecast points
Prepare input for hydrodynamic model based on hydrological
models, gauging stations, tidal boundaries
Run error correction model to update output form
hydrodynamic model
November 2010Deltares Configuration Course 76
FEWS External Modules
Available Model Adapters
CEH Adapters (SNOWP, SNOW, PDM, KW, ARMA, TCM) HR (ISIS) PlanB Adapters (TRITON & PRTF) DHI Adapters (Mike11, NAM) Midlands Region (DODO, MCRM) Southern Region (STF) Northwest Region (NW TF – Common Adapter) Wales (SW Overtopping module - Common Adapter) SouthWest (Bruton/Holbeam Dam module – Common Adapter) Delft (SOBEK, RIBASIM, HYMOS, Sacramento) SMHI (HBV) University of Karlsruhe (PRMS) JRC (Lisflood - PCRaster)
http://public.wldelft.nl/display/FEWSDOC/Models+linked+to+Delft-Fews
November 2010Deltares Configuration Course 77
General Adapter
General adapter concepts State handling ID mapping
November 2010Deltares Configuration Course 78
General Adapter -Concepts
Delft-FEWS
General Adapter
Module Adapter
Modulemoduleinput data
moduleoutput data
published interface
input data(pi xml)
Pre Post
output data(pi xml)
November 2010Deltares Configuration Course 79
General Adapter – Configuration
From global.properties
General Adapter keywords
DELFT-FEWS reserved keyword
November 2010Deltares Configuration Course 80
General Adapter – Export Activities
Export Activities
Time Series – scalar
Time Series – grids (map stacks)
Time Series – profile
States
ModuleDataSet
ModuleParameters
November 2010Deltares Configuration Course 81
Export Activities – Scalar/longitudinal time Series
. Exports time series from database to a PI-XML file
. Assign File name to export data to
. Assign Time Series Sets to export
. Written to exportDir
. LocationID’s & ParameterID’s will be translated on export (as indicated in IdMapping)
November 2010Deltares Configuration Course 82
Export activities - grid time series
. Same principle as exporting scalar time series
. Define name of export file
. Export file contains reference to grid
. Data exported as a “mapstack”
Files:
pcr00000.000
pcr00000.001
..
..
pcr00000.023
pcr00000.024
November 2010Deltares Configuration Course 83
Export activities - grid time series
Supported grid file formats
ASC: ArcView/ARC-INFO grid interchange file
Time Series as map stack
PCR: PCRaster file format
Time Series as map stack
BIL: USGS standard grid format
Time series in one file
November 2010Deltares Configuration Course 84
Export Activities – Module parameters
. DELFT-FEWS administers module parameters
. Exports to external module as PI-XML file
. Held in Config\ModuleParFiles
November 2010Deltares Configuration Course 85
Export Activities – Module dataset
DELFT-FEWS administers module dataset
Dataset: native module file Held in Config\ModuleDataSets by DELFT-FEWS as a ZIP file
Exports to external module by extracting data in ZIP file
Exported to a “root”directory
November 2010Deltares Configuration Course 86
Exporting States
DELFT-FEWS can manage module states
What is a state?
Module initial conditions
Handled in “native”module format
Type of state;
Cold State – default initial conditions
Warm State – initial conditions from a previous run
For each module requiring management of states;
ZIP file of cold state MUST be available in ColdStates dir.
November 2010Deltares Configuration Course 87
Exporting States
Defining “how” to export state
moduleInstanceId
stateExportDir
stateConfigFile
stateLocations
read write
From the perspective
of the model!!
November 2010Deltares Configuration Course 88
Exporting States
stateConfigFile
For export: Written by DELFT-FEWS ; read by adapter
For import:Written by adapter ; read by DELFT-FEWS
November 2010Deltares Configuration Course 89
Exporting States
. Defining “what” to state export; stateSelection
. State selection determines length of module run!!!!!
. Cold State: Always use a cold state to initiate module run
. Warm State: Use most suitable/recent state in search period
November 2010Deltares Configuration Course 90
04:00 16:00 04:00 16:00 04:00
• Start 02:00 Jan 05. T0 16:00 04 Jan
• Start 7:30 and T0 07:00 Jan 05• Start 19:30 and T0 19:00 Jan 05
• Start 07:30 and T0 07:00 Jan 06
• Start 02:00 Jan 06. T0 16:00 05 Jan
States
02:00 02:0007:00 19:00 07:00
State Search period
November 2010Deltares Configuration Course 91
General Adapter – Execute Activities
. Running modules
. Executables or Java classes
. Arguments can be passed
. Timout is set in case module “hangs”
. If Diagnostic file indicates error – or non zero return code
Executable considered to have failed
November 2010Deltares Configuration Course 92
General Adapter – Import Activities
Import Time Series
Scalar Longitudinal Grid
Import States
November 2010Deltares Configuration Course 93
General Adapter – Import Activities
. Importing time series
. Same principle as exporting
. File name & Time Series Set to save data to
. IdMapping used to translate locationId’s and parameterId’s
. Grids imported as mapstack + XML file
November 2010Deltares Configuration Course 94
General Adapter – Import Activities
. Importing states
. Simple instruction where to find stateConfigFile
November 2010Deltares Configuration Course 95
Burn-in profiles
. Avoid “abrupt” shock to model on startup
. Mainly relevant to HD modules (stability)
Only applied when starting from a cold state
Identify start value in cold state Gradual “climb” to actual value
November 2010Deltares Configuration Course 96
Burn-in profiles
Burn-in at Temple Sowerby
November 2010Deltares Configuration Course 97
Startup/Shutdown activities
Clean up files from module
Startup – before running module Shutdown – after completing module
November 2010Deltares Configuration Course 98
Module diagnostics/debugging
PI-XML includes module diagnostics exchange
To be written by module/adapter
General Adapter Defines where diagnostics file is expected
Not all logging passed – use native module log files
External module log inserted into NFFS log
Most messages DEBUG level
Error/Fatal: System messages
Detecting module failure
. Error in diagnostics log
. non-zero return code
November 2010Deltares Configuration Course 99
General Adaptor - Exercise
Add unit hydrograph model
November 2010Deltares Configuration Course 100
FEWS Gridded Data
November 2010Deltares Configuration Course 101
FEWS Gridded Data
Fews supports the display of both irregular and regular grids
Available (common) Import types
ArcInfoAscii BIL BUFR CSV PI HDF5 NetCDF
November 2010Deltares Configuration Course 102
FEWS Gridded Data
Commonly used data types
Precipitation Soil Moisture Pressure Wind Temperature Wave height Surge
November 2010Deltares Configuration Course 103
FEWS Gridded Data
Data can be manipulated using transformation and interpolation modules
Cookie cutter Interpolation Transformation Data hierachy
November 2010Deltares Configuration Course 104
FEWS Gridded Data
November 2010Deltares Configuration Course 105
Gridded Data - Exercise
Import Gridded Data