© A

BB

Util

ities

- 1

-20

02-0

1; N

M-M

W-0

169/

UT

UA

/ND

F

Utility Data Warehouse

UDW 200

Mats Werner

UTUA/NDF

© A

BB

Util

ities

- 2

-

Data Warehouse for storage of all kinds of data

SPIDER Utility Data Warehouse

Data Warehouse

UDW 200

SPIDERSCADA/EMS/DMS

other Utility Information Systems

Presentation inReports & spreadsheets

Presentation inOne-line diagrams

Station A

B8 B9

C1T1 T2

B1 B2

B3 B4 B5 B6 C7

C2

B7

10.25.32

Presentation as Trends &Historical information

Data mining and otheranalysis tools

Vedlikeholdsplaner: Region Syd

Ny planLukk Detaljplan Hjelp

27 28 29 30 31 32 331 8 15 22 29 5 12Juli AugustNr Stasjon/LinjeAnl.DelÅrsak Status

96S432-196S530-296S532-196S541-1

HasleFlesakerLysebotnRendalen

10 G2G1F1G3

OmbyggingÅrlig RevisjonMekaniskUtskifting

I GangUtsattOK Reg.OK Land

Region/Område

Stasjon/LinjeAlle

Region Syd Type

Spenning

Status

1996

Alle Alle

300kv UkerSkala

Utskrifter

Based on...

© A

BB

Util

ities

- 3

-

UDW – Utility Data Warehouse

Oracle-based Data Warehouse for Utility data Able to store the continuous inflow of data from the power process

Able to store information from any other Utility Information System

Makes all this data available for Data Mining.

High User Performance

High Availability

High Redundancy

© A

BB

Util

ities

- 4

-

UDW – Utility Data Warehouse

Utility Data Warehouse

OracleRDB

ERP

AMS

BMS

CIS …

Non-real-time Information Systems

Both Utility-wide Data Warehouse and on-line Historian!

SCADA

EMS DMS

Real-Time Network Management Systems

Historical Analysisin Control Room Environment

Decision Support in Office Environment

© A

BB

Util

ities

- 5

-

Utility Data Warehouse – Original Requirements

RDBMS supporting the latest SQL standard ”Ad Hoc” queries, ”Data Mining”, etc.

Continuous operation Availability > 99.96 % Redundancy Continuous inflow of data (every 10 sec, 24h/day)

Response times (incl. picture presentation): 3-5 seconds Extremely large database tables: > 500 million rows Continuous calculations

sum, max, average etc. Complex calculations using MatlabTM

Automatic archiving of data that should not be retained on-line On-line backup Automatic maintenance of the database

eliminates the need for a DBA at the customer

© A

BB

Util

ities

- 6

-

Relational Analysis The ability to analyze relations between data is the basis for Decision

Support and Data Mining

Relational Database Design makes this easy

Object ID

Time Stamp

Value Quality

123456 12:34:10 123.456 1

789123 12:34:10 2.234123 1

456789 12:34:10 213.4324 1

123456 12:34:20 8.456 2

789123 12:34:20 2.2455 1

456789 12:34:20212.143

1

…

Object ID

Object Name

Station

ID…

123456 TRF1 446

789123 TRF2 226

456789 BAY1 791

987654 BAY2 446

…

Station

ID

Station

NameLocation …

446 City23 Downtown

226 FOO WestRing

…

…

Example: ”Give me the name and the location of all lines that has been in alarmed state during the weekend ”

© A

BB

Util

ities

- 7

-

UDW - Open but integrated in SPIDER concept

UDW is a completely open system

UDW is also a well integrated product in the SPIDER concept

… as shown in the following areas: Open Commercial Tools and Interfaces

Application Programming Interface, API

Well Integrated in SPIDER

Powerful Database Structure

UDW as an integration platform

© A

BB

Util

ities

- 8

-

UDW - Open Commercial Tools and Interfaces

Based on Oracle Oracle is an open commercial relational database

delivering ”the industry's highest performance, reliability and security for thousands of popular applications”

All tools applicable for Oracle is also applicable for UDW Possibility to access UDW data directly in Oracle, using SQL or Oracle tools New development in Oracle means increased value of UDW

Open interfaces ODBC/JDBC Microsoft COM SQL*Net UDW API (see below)

© A

BB

Util

ities

- 9

-

UDW Application Programming Interface, API

Simplifies access to UDW data From C, C++, Java or Microsoft COM-based applications

Simplifies the creation of custom user applications Functions for inserting and reading many data points at once General browser routines, to access data without knowing the data model An Oracle-independent general database server service,

for remote SQL access without SQL*Net Implements Additional functionality

User-friendly access routines for setting and getting a range of time-series data

When data is stored by event, the data is retrieved as if it was stored cyclically Implements support for Daylight Savings Time, Seamless Data Retrieval and

Data Sets (as described later in the presentation) The UDW API is used when:

accessing UDW data in the SPIDER environment accessing UDW data through the Microsoft COM interface accessing UDW data in dedicated programs (including UDW-internal)

© A

BB

Util

ities

- 1

0 -

UDW - Well Integrated in SPIDER

UDW data can be presented in SPIDER pictures, on-line diagrams

the report and trend tools used in SPIDER

The object oriented Data Engineering for the UDW is integrated with SPIDER Data Engineering.

The UDW programs are monitored by SPIDER

Alarms and warnings from UDW are sent to SPIDER alarm list e.g. if the data sampling is not functioning properly due to network or hardware

problems

© A

BB

Util

ities

- 1

1 -

UDW - Powerful Database Structure

Data is stored in a user-friendly way Uncompressed in open Oracle tables

Database views to simplify common queries

Data model is temporal history of each object exist in the database

calculation algorithms related to certain objects are time-stamped

Network relations and calculations are therefore correct even when studying old data, e.g. data restored from off-line archives

Very good performance regardless of system and database size, By the use of the Oracle partitioning technique

Sampling schemes are easily defined using Object Types and Patterns

© A

BB

Util

ities

- 1

2 -

UDW – Object Types and Patterns

Sampling schemes defined in Object Types Each Object Type corresponds to one Oracle Table

An object to be sampled refers to a Pattern A Pattern refers to a number of Object Types

Enables easy set-up of sampling schemes for sampled objects

Objects to be sampled

10 sec measurands

indications by event

state estimator result

1 min average measurands

hourly minimum measurands

daily maximum measurands

sampled Object Types calculated Object TypesMeasurement pattern 1

Measurement pattern 2

Indication pattern

UDW patterns Object Types

For each Object Type is defined: Sampling method

cyclically – including sampling frequency

by event – with our without integrated dead band

Retention period

If it shall be archived when retention period is over

© A

BB

Util

ities

- 1

3 -

UDW as and Integration Platform (example)

Developing a simple application in MS Excel, using the UDW COM interface

Combine the data in the Utility Data Warehouse with the functionality of Excel:

•Visual Basic programs•Excel spreadsheet functions•Excel Charts

Platform for application development and cross-platform integration

© A

BB

Util

ities

- 1

4 -

UDW - Data Model

Sampled data is stored in Object Types A simple Power System model is also provided

including e.g. stations, sub systems, point classes To enable queries and selections based on relational criteria

e.g. ”show all measured values included in station named AMHERST”

Automatically populated from SPIDER Data Engineering Temporality is provided

Additional data relations possible to define easily Since an open relational database technology is used

Station A

B8 B9

C1T1 T2

B1 B2

B3 B4 B5 B6 C7

C2

B7

10.25.32

Real-world objects

10 sec measurands

Information and relations between real-world objects

1 hour averagedaily maximum

ObjectsStations

Sub-systems

Object Types: values related to sampled real-world objects

© A

BB

Util

ities

- 1

5 -

UDW - Data Sampling

Virtually any type of information can be stored in the UDW The actual sampling is highly dependent on the source system from which the data is

extracted.

SPIDER SCADA/EMS/DMS Sampling All types of data in the SPIDER real-time database Avanti can be sampled and stored in

the UDW database. Cyclically, according to a user-defined sampling rate. “By Event" (upon change), with or without an integrated dead band

Sampling from other Information Systems There are several ways to store data into UDW:

Direct SQL insert, e.g. through database links (assuming an RDB source) Direct access through open interfaces (e.g. ODBC, JDBC, Microsoft COM) Dedicated program that uses the UDW API Dedicated program that uses the ”bulk data inserter” (as used in SCADA-sampling) Access through Avanti (e.g. by first inserting data into Avanti)

© A

BB

Util

ities

- 1

6 -

UDW – By Event sampling, integrated dead band

Sampling data By Event with an integrated dead band a threshold value in percent of the measurement range is defined

measured value changes are accumulated between samples

if the actual value differs only slightly from the latest stored, it will thus eventually be sampled anyway

© A

BB

Util

ities

- 1

7 -

UDW - Quality Marking

UDW handles quality information together with all sampled values. propagates automatically through all calculations

validity of calculation result depends on quality of included values according to what is defined in Data Engineering

Mapping of quality states is defined in Data Engineering For SPIDER sampling these mappings are set by default, but are configurable.

Possible to distinguish quality information generated in UDW and quality information originating from the SPIDER Avanti database.

Object Value Quality

123456 123.46 Invalid

789123 2.000 Corrected

456789 213.434 OK

Object Time Stamp

Value Quality

123456 12:34:10 123.46 Invalid

789123 12:34:10 2.000 Corrected in SPIDER

456789 12:34:10 5.00 Corrected in UDW

Object Time Stamp

Value Quality

22222 12:34:10 125.46 Invalid

33333 12:34:10 7.000 OK

Original quality value Quality value in UDWafter manual correction in UDW

Quality value in UDWafter calculation (a+b)

© A

BB

Util

ities

- 1

8 -

UDW - Post Disturbance Review

The Post Disturbance Review, PDR, function allows the user to monitor the process behavior before, during, and after a disturbance

A disturbance recording can be manually triggered by an operator It is possible to playback the recorded disturbance data in a Single Line diagram. Upon operator request, data for completed disturbances can be saved on the off-

line medium used for archiving

Triggingevent(discarded)

time

Triggingevent

periodbeforeevent

period aftereventDead-

zone

Trigging event(prolonging)

In the DE tool it is specified: which data is to be monitored, which events that will trigger a PDR time periods the number of areas available for

storage of disturbance information

In the DE tool it is specified: which data is to be monitored, which events that will trigger a PDR time periods the number of areas available for

storage of disturbance information

© A

BB

Util

ities

- 1

9 -

UDW - Data Archiving

For each Object Type it is defined (in Data Engineering): how long it shall be retained on-line in the database if it shall be archived

When retention period has passed, data is automatically exported to the chosen off-line medium

UDW maintains a library system for the archived data. When data shall be restored the user specifies the type of data and for which

time interval the data shall be restored

Seamless presentation is supported If restored data and on-line data connects in time

Standard software can be used for the Archiving function. such as Legato NetworkerTM

© A

BB

Util

ities

- 2

0 -

UDW - Advanced Data Processing Powerful and versatile data processing capabilities

Times series calculations: Sum, Average, Max, Min

Snapshot calculations Any formula/calculation on objects for a certain point of time

Custom calculations Any formula/calculation on objects for many points of time

Matlab used as calculation engine All power of the Matlab tool is available for utilization Calculation wizard to ease simple calculation definitions

Complete ROLAP1 aggregation support (configurable) Data can be automatically aggregated in all dimensions

E.g. summary for e.g. a station or the whole power network Raw data and processed data are stored and handled the same way,

i.e. both can be used for e.g. data mining Automatic recalculation of all2 formulas

Both during sampling and at manual entry

Quality coding of all values Propagates through calculations

Calculation patterns (“templates”) For easy connection of sampled objects to a set of pre-defined calculations

and sampling profiles

1ROLAP = Relational On Line Analytic Processing

1ROLAP = Relational On Line Analytic Processing

2Restrictions apply on custom calculations to prevent recursive deadlocks

2Restrictions apply on custom calculations to prevent recursive deadlocks

© A

BB

Util

ities

- 2

1 -

UDW - Report and user interaction functions

UDW data is easily accessed from various client tools

There are four main access routes: Through SPIDER (historical data originating from SPIDER can be presented in

SPIDER trends and spreadsheet reports, and in one-line diagrams).

Direct SQL read (through SQL*Net, Oracle tools, Microsoft Query etc.)

Direct access through open interfaces - ODBC, JDBC, COM, (e.g. Microsoft Office programs).

Through the API or Java API (dedicated programs accessing UDW data)

Database(Oracle)UDW 200

Presentation inReports & spreadsheets

Presentation inOne-line diagrams

Station A

B8 B9

C1T1 T2

B1 B2

B3 B4 B5 B6 C7

C2

B7

10.25.32

Presentation as Trends &Historical information

Data mining and otheranalysis tools

Vedlikeholdsplaner: Region Syd

Ny planLukk Detaljplan Hjelp

27 28 29 30 31 32 331 8 15 22 29 5 12Juli AugustNr Stasjon/LinjeAnl.DelÅrsak Status

96S432-196S530-296S532-196S541-1

HasleFlesakerLysebotnRendalen

10 G2G1F1G3

ImpingingÅrlig RevisjonMekaniskUtskifting

I GangUtsattOK Reg.OK Land

Region/OmrådeStasjon/LinjeAlle

Region Syd TypeSpenning

Status

1996

Alle Alle300kv UkerSkala

Utskrifter

© A

BB

Util

ities

- 2

2 -

The Time Tagged Trend (TTT) function is used for presentation of historical data in curve diagrams.

Up to 16 curves in each diagram each with its own value axis.

Automatic scaling of value axisdepending on curve values.

The values can be fetched from different object types Can have time offsets rela-

tive to each other.

Quality coding

UDW – Client example: SPIDER Time Tagged Trends

© A

BB

Util

ities

- 2

3 -

UDW data can be shown in the Single Line Diagram by selecting a historic time and an Object Type or a Data Set.

Possible to start a replay of the data using tape-recorder like controls: play, advance, rewind, stop the selected time is automatically changed with a given speed.

UDW – Client example: SPIDER Single Line Diagram

© A

BB

Util

ities

- 2

4 -

There are several ways to get UDW data into Excel: via the SPIDER Excel add-in directly through the

Oracle ODBC driver copy/paste or export/

import data retrievedwith another tool, e.g. Oracle™ Browser

use the UDW COM-interface in an ExcelVisual Basic program, that reads the approp-riate information from UDW

Once the data is in Excel, its built-in functions can be used i.e. data can be calculated on, charts can be created etc.

UDW – Client example: Microsoft Excel

© A

BB

Util

ities

- 2

5 -

Using the Microsoft Query tool it is possible to create custom queries to the UDW in a user-friendly way, from the Windows environment

UDW – Client example: Microsoft Query

© A

BB

Util

ities

- 2

6 -

Using the integrated MatlabTM server it is possible to define advanced formulas and calculations

Using a MatlabTM client tool it is also possible to benefit from the advanced graphical user interface of MatlabTM, and that way visualize data and corre-lations found in UDW data.

UDW – Client example: MATLABTM Graphics

© A

BB

Util

ities

- 2

7 -

UDW - Data Sets

The Data Set function facilitates presentation of UDW data

Groups a set of Object Types Can thereafter be referenced as one identity

A Data Set may include object groups from different SPIDER concepts: Measurands, Process values, Indications etc. Practical e.g. when presenting historical data in Single Line diagram

© A

BB

Util

ities

- 2

8 -

UDW - Seamless Retrieval – Data Set example

Measured values are often sampled with several frequencies, where higher frequency normally corresponds to a shorter retention period Each sampling frequency corresponds to one Object Type

For some points of time there may exist data in more than one Object Type

By creating a Data Set of these Object Types, it is possible to address the sampled values as one entity

get best possible resolution for all points of time

10 second

1 minute

1 hour t1 t2 t3 tn

Data that are retrieved when asking for data from t1 to tn:

t1-t2 1 hour values

t2-t3 1 minute values

t3-tn 10 second values

Data that are retrieved when asking for data from t1 to tn:

t1-t2 1 hour values

t2-t3 1 minute values

t3-tn 10 second values

© A

BB

Util

ities

- 2

9 -

UDW - Daylight Saving Time

The UDW has full support for Daylight Savings Time. It is possible to store both normal time and calendar time in UDW.

It is recommended for most usages to only store normal time and let the UDW API take care of the DST conversions.

If the times are retrieved through the UDW API, the UDW takes care of the conversion both when it comes to individual points of time and

larger time spans.

The calendar time is used in display pictures and reports

The UDW API is used when accessing UDW data in the SPIDER environment, by the MS COM interface and when accessing UDW data in dedicated programs

The UDW API is used when accessing UDW data in the SPIDER environment, by the MS COM interface and when accessing UDW data in dedicated programs

Calendar time is normal time with Daylight Savings offset

Calendar time is normal time with Daylight Savings offset

Example: If the user wants to get data for a time span that includes a switch between normal time and Daylight Savings Time, there will be the correct number of times retrieved (e.g. an extra hour if going from daylight savings to normal time).

© A

BB

Util

ities

- 3

0 -

UDW - Authority

UDW uses standard Oracle™ authority routines

Users are assigned to a Profile and one or more Roles Roles provides specific access rights

Profiles are used to set session parameters Such as the maximum CPU time for a session

Examples of Roles defined in UDW: system administrator, privileged user, common user, etc.

Profiles are used to limit the risk for low priority users or user groups to impair the performance of the UDW database by using badly formed SQL-statements, for instance, a full table scan.

User

Role

Number of sessionsMax CPU per session

Connect time during sessionetc.

Profile

System administrator

Database creator

© A

BB

Util

ities

- 3

1 -

UDW - Audit Trail

The Audit Trail function in UDW tracks all changes made in the UDW database

For all manual updates the following data is stored: old and new value the affected time and object identity the point of time of the modification an operator identification

The standard Oracle™ audit trail function can also be used providing tracking of user actions on a table level.

Tracking of insert is not recommended the insert rate is very high in the system, and auditing insert would degrade

performance.

© A

BB

Util

ities

- 3

2 -

UDW – Planned Values

The UDW supports storage of planned values A ”window” of future data points are a opened for storage Can be used to store e.g. forecasts or schedules

Defined as an Object Type in Data Engineering Handling of retention period, archiving etc. just like other Object Types

A number of parameters are possible to define: if the values shall be overwritten by actual sampled values or not if the values shall initially be set to blank or ”invalid” the size of the ”window” – the time span to be opened in the future initial time period of the plan, i.e. including historical times

© A

BB

Util

ities

- 3

3 -

UDW - Data Engineering

The Data Engineering of the UDW is object oriented and dialog driven.

Is totally integrated with the Oracle-based SPIDER data engineering tool. All data engineering logic for UDW is contained within the UDW product.

© A

BB

Util

ities

- 3

4 -

UDW – Specification of Object Types

For sampled Object Types: sampling method: by event (including dead band) or cyclically (including

frequency) retention period, i.e. how long the object shall be retained on-line in the database mapping between the source system (SCADA) and UDW quality codes properties to be sampled from source system (SCADA) to UDW

for instance the actual value and updated flag for a measured value

For calculated Object Types: source Object Type (a sampled Object Type or another calculated Object Type) type of calculation (e.g. Average, Sum, Max, Min) required percentage factor of valid source data to get a valid result if time stamping shall be in the beginning or end of the calculated period

© A

BB

Util

ities

- 3

6 -

UDW - Backup and recovery

For safety copy reasons the on-line data in UDW can be backed up, using standard Oracle™ functionality.

The backup is done in a way so that all data, including calculated values, are consistent.

The interval between backup activities is configurable

UDW Safety Backup and UDW Archiving are complementary functions!

UDWSafety Backup

UDW on-line system

Archiving of data older than retention period

UDW Archiving

Utility Data Warehouse

OracleRDB

Archiving of a safety backup of on-line data

© A

BB

Util

ities

- 3

7 -

UDW - Configuration

The UDW can be configured in different ways to provide the required degree of redundancy and data security.

Data Redundancy The data in the UDW database is stored redundantly so that if one disk fails, no data is

lost To get full data redundancy, RAID disks of type 0+1 are used

striping of data and disk mirroring is applied

Server Redundancy The UDW servers are redundant so that if one UDW server becomes inoperable (e.g. due

to hardware errors) another server is taking over.

Different configurations are applicable depending on the requirements on allowable downtime and loss of data during switch-over. Redundant Server solutions, with automatic switchover Emergency Center solutions, with manual switchover Single Server solutions

© A

BB

Util

ities

- 3

8 -

UDW - Redundant Server solution

Redundant Server solution No single point of failure

Automatic switchover when the on-line server malfunctions

Separate server for Safety Backup and Archiving

Memory Channel

UDWServer 1

RAID Disk

Server-RAID Server-RAID

Connection

SCSI

Connection

SCSI

DLT Automated Tape Library with duplicatedrives for archiving and restoring

LAN

SCADA/EMS/DMS

SCSI

UDWServer 2

Backup Server

SCSI

Backup LAN

Memory Channel

UDWServer 1

RAID Disk

Server-RAID Server-RAID

Connection

SCSI

Connection

SCSI

DLT Automated Tape Library with duplicatedrives for archiving and restoring

LAN

SCADA/EMS/DMS

SCSI

UDWServer 2

Backup Server

SCSI

Backup LAN

The two UDW servers are connected in a cluster configuration

Both servers receive data to be stored from the sampled system.

One of the servers is on-line while the other is hot standby and ready to take over the processing.

The on-line server updates the UDW database with the sampled data and calculation results

© A

BB

Util

ities

- 3

9 -

UDW – Emergency Center solution

Emergency Center solution No single point of failure

Manual switchover when the on-line server malfunctions

Original UDW rebuilt by restoring a safety backup

The standby server is kept up-to-date with the on-line server, using Oracle Standby

It is possible to restart the standby database as the primary UDW database with minimal loss of time and data.

SCADA/EMS/DMS HIS Server, online

LAN

WAN

SCADA/EMS/DMS HIS Server, standby

LAN

RAID Disk

Server-RAID

ConnectionSCSI

RAID Disk

Server-RAID

ConnectionSCSI

Em

erg

ency

Co

ntr

ol C

ente

r

© A

BB

Util

ities

- 4

0 -

UDW – Single Server solution

Single Server solution With either RAID or disk shelf

If a disk shelf is used instead of RAID, the single disk controller is located in the UDW Server

SCADA/EMS/DMS

SCSI

UDW ServerDiskShelf

LAN

or RAID