Ilya SlutskerData Base Development, Processing and Data Archive

• Data Collection• Archives• Processing• Data Access Delivery and Distribution

Data Collection

Historically, first data from Automatic Sun Photometers were collected manually. Starting in 1993 DCP (Data Collection Platforms) have become preferable way of getting data.

Today, about 92 % of data is collected using automatic DCP transmissions, and only about 8% is send using manual dump of the instrument memory to PC and submission of this dump to Aeronet server.

Main reasons for the manual downloads are 1) Unavailability of the direct satellite uplink – Far South and North, Landscape profiles 2) Local regulations. 3) Equipment malfunctioning. Most of those problem eventually can be remedied by using cellular or digital communication technology.

Until May 1993 We did not use DCP Satellites Uplinks

In 1993 we started transmitting using GOES East from US, Canada and BrazilSoon we extended our transmission coverage area to GOES West – Alaska, Hawaii …Meteosat gave us access to European, African and Near East sitesFinally by the end of the last century We covered Far East and Pacific using GMS

DCP Data Collection

DCPs transmit data and satellites retransmit them to Data collection Stations.These Stations receive data, demodulate them, sort them and then send to users.

Goes East and West data are processed by NOAA DAPS at Wallops Island, Virginia.

Meteosat are processed by EUMETSAT in Darmstadt, Germany

GMS data are processed by Vitel built receiver,Located at the NOAA PTWS, Ewa Beach, Hawaii

Even manual data download changed very much during recent years.Now Site Managers can submit data on-line 24 hours a day, and the data Will be automatically archived and processed within 1 hour

After being submitted to Aeronet serverData are processed and placed to several archives

DAPS EUMETSAT PTWS

Hourly Auto Upload

DCP Messages Archive

K7_upload.html

K7 Archive

Manual upload

Daily Reports Archive

Raw Data Archive

DCP decoding K7 decoding

Processed Data ArchiveProcessing Software

We maintain several archives that allow us properly process the data.

• Filters Archive. It has spectral response function for filters installed in our instruments. Now the information about more that 3000 filters is stored there.

• Sites Archive. All information of Aeronet sites where instruments have been operated. Currently 496 Sites.

• Instrument Configurations Archive. Many instruments have their configurations changed over years. The archive keeps that information including types of instruments and filters used.

• Calibration Archive. Sun , Sky and Flux calibrations for all instruments.

We use the program “administrator” to manage this archive.

Filter Functions are sorted first by nominal channel (i.e. 1020, 440 …)Then by the year of shipment (1997, 1998 …)Then by the name of batch (1020_1, Lot2401 …)And finally by the number of the filter.

Filters Archive

Sites Archive

Sites are sorted first by types : Permanent, Seasonal, Temporary,by Region and Country/State, then by PI’s name.

We can search sites using all the above parameters and alsoby the coordinates and name of the site.

The archive is managed by“administrator”

There have been more than 20 different instrument types over years. Most common are :Standard, Polarized and Advanced. Each type characterized by its own set of filters.

The Archive is managed by “administrator”.

Instruments Configurations Archive

Calibration Archive

This archive is managed by “demonstrat” program. Example – Calibration history.

Data Archive

• This is where all the measurements data and Aerosol Property products are kept.

• There are 2 archives : “Raw Data Archive” and “Processed Data Archive”.

Raw Data ArchiveProcessed Data ArchiveProcessing

Software

Direct Solar Measurements

Sky Measurements (Almucantars, Principal Planes, BRDF)

Fluxes from Flux sensors

Temperature, Internal and External battery voltage, Dark current, resets, hardware statuses and errors

Aerosol Optical Depth

Water Vapor

Sky Radiance retrieval products ( Size distribution, Phase Function, Single Scattering Albedo, Refractive Index …)

Quality Assured AOD

Processing Sequence and Software

Once the data are collected and archived in the Aeronet server, several processing tasks are performed.

Each of these tasks is managed by its own software.

1) Individual raw solar measurements are used and level 1 AOD and Water Vapor are computed. Runs every hour.

2) Daily blocks of raw solar measurements are used and level 1.5 cloud screened AOD and Water Vapor are computed. Runs every day.

3) Individual Almucantars and Principal Planes and Daily blocks of Level 1.5 AOD are used and Sky retrieval products are computed. Runs every day.

All these tasks run automatically After they are performed, the data become available to our group. Normally all

the data from the previous day are available by the noon of the current day. The data can be reprocessed, due to 1) recalibration, 2) coordinates adjustments

3) quality assurance. Reprocessing programs are started by the authorized user and reprocess long records of data , sometimes months and years per run.

Data Access

All the data are accessible through several interface On the System Level we have access to the binary files which can be backed up,

cloned or, if necessary deleted. There are encoding and decoding programs that can retrieve the data into ascii text, that can be looked at, modified, and saved.

On the User Level users have access to “demonstrat” , which is data browsing tool. They can look at the data, download the data, and perform several research task. Authorized users can calibrate devices, reprocess data and raise the quality level of data.

On the Public Level everybody have access to the Web Site where they can look at the data and download them.

Our goal is to bring as much data as we can to the Public Level limiting Public Access by only actually changing the data. Only Authorized users are going to be able to modify the data. As for the raw data, they are locked and not modified.

Next slide will show some example of the “demonstrat” use.

Demonstrat Main Window

We can click on almost anything,And also drag the mouse across space and time.Right click on Site or Instrument number will limit selection toThat site or instrument.

AOT Time Dependence Window

Langley Calibration Window

Almucantar Window

Conclusion

• As the network of instruments and sites grows and now scattered across the world with no continents to spare, the reliability and performance of data storage and processing software have become of the essence.

• Over the years we developed the system which collects, archives, processes and distributes Aeronet data from hundreds of sites and instruments and does it fully automatically in real time.

• Although all the routine processing and storage is done automatically, we keep complete control over the system performance and can monitor and troubleshoot it at any moment.

• The system development continues, as new data type and products become available.