Climate Data Management System

CliSys

U. R. Joshi. National Data Centre

India Meteorological Department Pune

Outline of Presentation

• NDC Technological Up-gradation

• CliSys Features

• System Architecture

Database

Metadata

Data Acquisition

Data Generation

Data Production

Technological Up-gradation at NDC

1906 - IMD HQ was shifted from Kolkata to Shimla

1928 - IMD HQ shifted from Shimla to Pune

1942 - All met. data transferred to Pune

1945 - Transfer of data onto punch cards started

1977 - Third generation Computer EC-1040 was installed

1977 - National Data Centre (NDC) came into existence and data from punch cards were transferred to Magnetic tapes

1986 - VAX 11/730 Computer System installed

1993 - VAX - 4000/ 300 Computer System installed

2001 - Data from magnetic tapes were transferred to CDs/DVDs

2002 - Compaq Alpha Server was added

2010 - Climate Data Management System (CliSys) installed

2011 - All Historical data loaded into CliSys Data Base

Manuscript

CLISYS System

Desktop Computer CD’S & DVD’s

Manuscript Punching Machine

CliSys System

Punch Card

Card Reader EC-1040 Computer Magnetic Tape VAX 4000/300 Computer

CDs/DVDs

Transformation Of Technology

CliSys

• It provides a set of tools and procedures that allow all data relevant to climate studies to be properly stored and managed by:

• Ensuring the collection and storage • Data flow monitoring by Geographical and Statistical

view. • Automatic Data Processing by tolerance value and

consistency controls. • Easy access to climate data with web based interface • Providing tools for analysis and statistical

interpretations • Creating customized data products

Under VARSAMANA Project, a Climate Data Management System (CliSys) was acquired from Meteo France International (MFI)

CliSys: An Overview CliSys Follows WMO recommendations:

Ascribes each data value with a comprehensive quality flag Defines and configures easily data controls Offers an historic on data modification, Ensures data security (archive and user right policy), etc.

CliSys System Architecture

CliSys Web Functions and Menus

CliSys Consistency and Integrity Controls

CliSys: Database Structure

CliSys database is divided into 5 main Schemas:

•A Metadata Schema META

•A Data Schema DATA

•A Climatological Operation Schema CLIMOP

•An Administration Schema CLISYS •A Web Application Schema CLISYSWEB

The Metadata model: More than 50 tables to manage stations, instruments, parameters, etc.

CliSys: Metadata Structure

• Station Management • Parameter Management. • Address Management • Owner Management • Instrument Management

CliSys: Data Acquisition

CliSys: Data Generation

CliSys: Quality Control

• In CLISYS, controls to qualify each data are mainly designed for hourly and daily data

• The doubtful data then needs a climatological review by the operator and the updated data is considered for next level of data checking

• The CLISYS quality controls pass through 4 levels:

1. Syntax + Tolerance Tests

2. Priority Test

3. Filter tests: Oracle Table Constraints and

4. Consistency Controls

CliSys: Quality Control…

• “Meteorological” controls are performed once the data are in the database

• Some of the controls available in CLISYS are

• Global Limit Check

• Related Elements Global Limits

• Internal Consistency

• Global Rate of Change and Station Record Limits.

• Standard Deviation Check

• Station Rate of Change Check

CliSys Quality Flags: 2-Digits

X : Has the data been checked? If yes what is the result of the control?

The flag is named by the following label : Fxxx where xxx is the name of the flagged parameter.

0 raw data 1 calculated data using only raw data 2 modified data, not a raw data and not calculated by CliSys 3 calculated data generated with at least 1 non-raw data

1 non-checked 2 checked data, declared doubtful by the automatic process 3 protected data, data that never passes through the controls (human decision) 4 checked data, validated by the automatic control process

Y : Is the data a raw data? If not is it a calculated data or a modified data?

XY used to indicate Quality Flags for each Parameter

Hardware and Software

Front-Office Server – II

Front-Office Server – I

Post-Processing Server

Training Server

Development Server

Supervision Screen

SAN

Tape Library

Supervision Screen

Synergy -I

Synergy -II

Standby Acquisition Server

Operational Acquisition Server

Supervision Screen

Standby Database Server

Operational Database Server

Rack 1 Rack 2 Rack 3

Installed Servers

CliSys System Installed in IMD Pune

Logical View of CliSys System:

Servers Model RAM Disk space

Database HP ProLiant DL580 Intel Xeon E7340

Processor speed 2.40 GHz 16 GB 292 GB

Application

HP ProLiant DL580 Intel Xeon E7330

Processor speed 2.40 GHz 4 GB 292 GB

Acquisition

Production

Web server

Backup HP ProLiant DL380 G5 Intel Xeon X5450

Processor speed 3.00 GHz 4 GB 292 GB

Work Stations

(17 nos.) HP DC5800 Processor speed 2.20 GHz 2 GB 240 GB

Operating System (OS) CENTOS 5.4

Database Oracle 10g

Climate Data Management System CLISYS ver. 7

Interface CLISYS WEB

Workstation OS Windows XP

CliSys Hardware:

CliSys Software:

CliSys Hardware and Software

Metadata Fields

Few Administration Tools

Online Data Monitoring

Online Data Acquisition Monitoring

Offline Data Loading

Loading Supervision…

Data Constraints while Loading

Station Description / Selection

Data Access

CliSys Computed Normals

CliSys Data Products

Climatic Data

Time Series

Wind Rose

Temperature Contours

Temperature

Wind Rose Diagrams

Bar Diagrams

Thank You