Page 1 Envisat Validation Workshop, Campaign Database, 12/12/2002 Envisat Validation Workshop...
-
Upload
shannon-williamson -
Category
Documents
-
view
214 -
download
0
Transcript of Page 1 Envisat Validation Workshop, Campaign Database, 12/12/2002 Envisat Validation Workshop...
Page 1Envisat Validation Workshop, Campaign Database, 12/12/2002
Envisat Validation Workshop
Atmospheric Chemistry Validation Team
Ground-Based Measurements and Campaign Database Subgroup
Campaign DatabaseTerje.Krognes, [email protected]
Aasmund.Fahre.Vik, [email protected] Norwegian Institute for Air Research (NILU)
Page 2Envisat Validation Workshop, Campaign Database, 12/12/2002
PresentationPresentation OutlineOutline
Data Centre StatusTools
Local end-user tools
WWW search tools
Commissioning Phase Activities
Page 3Envisat Validation Workshop, Campaign Database, 12/12/2002
Data flowData flow Signed protocol required
for access to tools, docs and files
ECMWF data requires additional protocolASC2HDF
SEARCH
HDF2ASC
OK?
DATA
DATABASE
table.dat
DOCSDATA ORIGINATOR
DATA USER
FILES
nadir web
ftp to zardoz
http
Error email
http or ftp
Page 4Envisat Validation Workshop, Campaign Database, 12/12/2002
Data Centre Status (1/3): HardwareData Centre Status (1/3): HardwareThe machines have been swapped
between production and spare serversThe hardware has not been changed
Firewallzardozftp,
telnet
http
Storage
InternetInternetspare
nadir
Page 5Envisat Validation Workshop, Campaign Database, 12/12/2002
Data Centre Status (2/3): SoftwareData Centre Status (2/3): Software Web-server (nadir) runs Linux Redhat 7.2 with
up2date ~weekly (6.2 at 2nd rehearsal) File server (zardoz) runs Solaris 7, as before Nusphere MySQL
Apache 1.3.22 (1.3.14) MySQL 3.23.44 (3.23.32) PHP 4.0.6 (4.0.4pl1)
IDL 5.4, as before HDF 4.1r3, as before No re-boot on nadir since April 2002 Uptime on zardoz since April 2002 is approximately
99,98% (2 half-hour maintenance + re-boot)
Page 6Envisat Validation Workshop, Campaign Database, 12/12/2002
Data Centre Status (3/3): MetadataData Centre Status (3/3): Metadata
Metadata Guidelines updated by metadata board on 24 April 2002
During commissioning phase, updates of various metadata table entries, without affecting structure: Project-related information
Actual descriptors
Guidelines still applicable, but more clarifications to be added.
Page 7Envisat Validation Workshop, Campaign Database, 12/12/2002
New user toolsNew user tools Major re-write of the ASC2HDF file generation and
checking tool around launch 6 smaller upgrades of the ASC2HDF package after
launch Entire ASC2HDF source code (FORTRAN) published to
users. HDF output routines may be combined with user’s own input and processing modules
IDL-tool to generate HDF data files (without error checking): IDL2HDF
IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally
IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)
Page 8Envisat Validation Workshop, Campaign Database, 12/12/2002
New web functionality New web functionality - ECMWF- ECMWF
ECMWF plots (NH 2,5º*2,5º and T106)
ECMWF 10-day back trajectory plots
T106 extractions to HDF files
Page 9Envisat Validation Workshop, Campaign Database, 12/12/2002
New web functionality - search and plotNew web functionality - search and plot
Use of defaultsFacility to save and retrieve a searchProvision for downloading multiple files in tarball via web
interface Improved on-line plotting (also for multidimensional data)Corrected display of comments to variables
Page 10Envisat Validation Workshop, Campaign Database, 12/12/2002
File searching (1/4)File searching (1/4)
Defaults are marked with *** Page initially opens without
defaults Page may be reset or set to
saved defaults bys selecting a previously saved query
Files may be filtered by a large number of metadata parameters
Page 11Envisat Validation Workshop, Campaign Database, 12/12/2002
File searching (2/4)File searching (2/4)
The default criteria page lists all criteria sets that this user has previously saved
Entries may easily be removed The main search page will be
loaded with the selected defaults when launched from this page
Defaults may include 4D space-time parameters and variable name wild-cards
Page 12Envisat Validation Workshop, Campaign Database, 12/12/2002
File searching (3/4)File searching (3/4)
The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page
Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists.
Variable name and variable mode accept multiple selections
4D space-time criteria may be entered as a single point, or as a max/min box
All files that include or touch the 4D criteria are listed as matches
Page 13Envisat Validation Workshop, Campaign Database, 12/12/2002
File searching (4/4)File searching (4/4)
Files may be selected by posting date
Files updated after a given date may be selected
The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID
If no criteria are selected, the entire file list will be requested
Page 14Envisat Validation Workshop, Campaign Database, 12/12/2002
File ListFile List The file list displays the current
selection criteria A single file may be downloaded
immediately Files may be marked and
downloaded in a compressed tarball
Search criteria may be saved for later re-use (type a meaningful description before saving)
The Variables link brings up a variable list and other documentation contained in the file
Page 15Envisat Validation Workshop, Campaign Database, 12/12/2002
Variable ListVariable List The variable list page displays all
parameters contained in a file Variable name, mode and descriptor are
displayed The measurement unit and
corresponding SI unit are displayed All documentation fields attached to the
file are displayed Additional documentation fields attached
to a single variable may be displayed through the Comments link
Variables may be selected to generate an on-line plot (next slide)
Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots
Page 16Envisat Validation Workshop, Campaign Database, 12/12/2002
Variable plotting (on-line browse)Variable plotting (on-line browse)
Page 17Envisat Validation Workshop, Campaign Database, 12/12/2002
Administrative toolsAdministrative tools
Web tools to report uploads/downloadsWeb tools to archive/retrieve outdated
data filesWeb tools to check and maintain index
database consistency and integrity
Page 18Envisat Validation Workshop, Campaign Database, 12/12/2002
User activitiesUser activities Users have sent approximately 200 emails with requests for accounts
or updates to metadata, or with questions to HDF file generation After launch focus has slowly shifted to requests for help with HDF file
formatting issues and removal of outdated files From May to November 2002 users have submitted 5880 HDF data
files (570 of these were rejected due to formatting problems, and 238 have later been archived).
From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included)
There are currently 4741 ACVT files open to users (22 active AOIDs) There are currently 462 MAVT files open to users (5 active AOIDs)
Page 19Envisat Validation Workshop, Campaign Database, 12/12/2002
User activities: Up and DownloadsUser activities: Up and Downloads
0
200
400
600
800
1000
1200
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
uploads
downloads
The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz
Page 20Envisat Validation Workshop, Campaign Database, 12/12/2002
Nadir-team activitiesNadir-team activities 300+ interactions with users (of which 50% during Commissioning Phase) 250+ interactions with ESA and the Metadata Board (10% during
Commissioning Phase) One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs Maintained and improved shell scripts for automatic file checking and storage Created additional IDL and FORTRAN tools Performed extensive web program development Performed user administration updates Added metadata entries and updated the table.dat file 23 times since launch Developed and maintained scripts and data flow to provide pre-processed
ECMWF data extractions and plots on the web Monitored the automatic file reception system (now containing nearly 5000 files
totalling 275 MB + index database 20 MB)
Page 21Envisat Validation Workshop, Campaign Database, 12/12/2002
Metadata issuesMetadata issuesAlthough many quality checks are performed on the data upon submission to
the database: The metadata definition is currently not complete enough to perform
automatic checks of the validity of reported measurement units Some components and units may still be inconsistently named or mis-used
Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.