Argo data management
28
Argo 4th Scientific workshop Venice September 2012 1 Argo data management
Transcript of Argo data management
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
1
Argo data management
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Real Time Argo Data System
Eleven National Data Centers AOML, BODC, Coriolis, CSIO, CSIRO,
INCOIS, JMA, KMA,KORDI, MEDS NMDIS
Two GDAC: Coriolis –US-Godae
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
3
Argo real-time quality control
Profiles : 17 automatic QC tests performed before gdac and gts distribution
1 Platform Identification2 Impossible Date3 Impossible Location4 Position on Land5 Impossible Speed6 Global Range7 Regional Range8 Pressure Increasing9 Spike10 Top and Bottom Spike : removed11 Gradient12 Digit Rollover13 Stuck Value14 Density Inversion15 Grey List16 Gross salinity or temperature sensor drift17 Visual QC (not mandatory)18 Frozen profile19 Deepest pressure
Trajectories : 7 automatic QC tests performed before gdac and gts distribution
1 Platform Identification2 Impossible Date3 Impossible Location4 Position on Land5 Impossible Speed6 Global Range7 Regional Global Parameter
ALL measurements are provided to users. Use the flags to
know if the data are good
To be on the safe side only use data with flag=1
QC flag scale0 No QC was performed
1 Good data2 Probably good data3 Bad data that are potentially correctable
4 Bad data5 Value changed6 Not used7 Not used8 Interpolated value9 Missing value
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012 Real-Time QC can't detect all anomalies !!
Realtime QC automatic tests thresholds are a compromise between letting bad data going through and stopping good data
Profiles are mainly qualified one by one and the experience has proven that individual profiles may look good while they are not coherent with the neighbours
4
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Delayed Mode Argo Data System
12 Delayed Mode Operators BODC–BSH- Coriolis- CSIO-CSIRO- INCOIS- JAMSTEC -
OGS- PMEL- UCSD –WHOI - UW
Six Arcs coordinated by AOML- BODC Coriolis –OGS –INCOIS – IPRC
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Example: WMO ID 3900161
Salinity drifted salty over time (blue drifting to red). sensor drift
over time
Delayed mode processing to detect Offset or Drift
Comparing with high quality CDT in the area using statistical method allows to estimate the correction to be applied
(Wong & Owens 2009)
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Reference data base for delayed mode QC : A need for more recent CTD as the Ocean is changing !
Coordinated by Coriolis (C Coatanoan) Collaboration with Clivar/CCHDO NODC/USA and PIs
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Reference data base for delayed mode QC : A rgo good profile data base
8
for poorly CDT sampled, high variability or complex area
Coordinated by UCSD (J Gilson)
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
9
Argo GDAC global FTP servers Monterey & Brest
The GDAC FTP sites provide the master copy of Argo data set(meta-data, profiles, trajectories and technical information).The GDAC keeps the highest processing level of floats data.It is not in charge of archiving the different versions of float data.
http://www.usgodae.org/argo/argo.html http://www.coriolis.eu.org/cdc/argo.htm
ONLY GDACS GUARANTY ACCESS TO THE BEST VERSION OF FLOAT DATA
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Periodic monitoring for an enhanced Argo dataset
The Goal: Detect anomalies earlier than Delayed Mode QC
if possible before the users !
10
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Process of quality control
database
Argo data submissionFlow of raw data
ctd, xbt, …
Detection of anomalies
Objective analysis
Message to each dac when a flag has been updated in the Coriolis database
AOML BODC CSIOCorrection in netcdf files
Comparison with neighbors: Objective Analysis at CORIOLIS GDAC
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
12
Last Month map : about 100 profiles among 10 000 detected
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Comparaison to Altimetry
• As, Sea Level Anomalies (SLA) from altimeter measurements and Dynamic Height Anomalies (DHA) calculated from in-situ T and S profiles are complementary but also strongly correlated Satellite altimeter measurements are used to check
the quality of the Argo profiling floats time series
• The main idea is to compare co-located SLA and DHA to detect systematic or punctual errors in the Argo data sets time series
• Analysis are performed for each float time series
*S. Guinehut, C. Coatanoan, A.-L. Dhomps, P.-Y. Le Traon and G. Larnicol, 2008: On the use of satellite altimeter data in Argo quality control, accepted in JAOT.
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Float : 3900133r : 0.91rms-diff : 20.44 %mean-diff : -0.73 cmsamples : 147
Validate the consistency by comparing to altimetry
Stephanie Guinehut – CLS - FRANCE
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Float : 1900249r : 0.00rms-diff : 1538.0 %mean-diff : -8.98 cmsamples : 152
• Progressive drift of the salinity/pressure sensors :
Validate the consistency by comparing to altimetry
Stéphanie Guinehut Poster
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
0
20
40
60
80
100
120
140
Anomalies
Feedbacks
Status on the anomalies detected since June June 2008
AnomaliesFeedbacks
On the 25 anomalies detected in 2012, • 12 pop up in March , 2 in June and 5 in September• 6 are old ones that need to be corrected
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Known anomalies that have affected the Argo dataset in the past years
17
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Known anomalies that have affected the Argo dataset in the past years
Incorrect pressure binning on some FSI and SBE SOLO Floats deployed by WHOI in 2006: the data with incorrect or doubtful depth have been flagged since November 2008
The Truncated negative pressure offset on APEX floats with APF8 controller: Floats have been identified and data corrected in delayed mode. The problem doesn't exist on APEX since APF9 controller (better since 2008)
The SBE pressure sensor microleak : The problem doesn't appear on floats deployed since 2010 as the floats have been retrofited by SBEWhen the leak is important , the pressure offset is important and the data are flagged as bad 18
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Pressure correction – agreement between CSIRO audit/DAC In August 2012
CSIRO monitor 2 times a year how well metadata identifies float type and whether the agreed pressure correction is applied correctly
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Impact of TNDP on total data set: based on CSIRO audit
• Reduction in TNPD as APF8’s die out and towards recent years as only reported in DM where sufficient data exists
•TNPD degraded 20% of profiles from 2003-05, proportion decreasing
• overall ~10% of total data set TNPD affected
• great improvement by DACS since 2008 when 30% of data was affected
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
What’s new since last Argo Science Meeting in China in 2009
21
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Improvement of Trajectory dataset
Improve the quality and consistency of data that is currently in trajectory files
Timing Calculation Position calculation and Real Time QC on position Metadata information Grounding information
Enhance the trajectory file format to include more useful and accurate information to enable easier velocity calculations
Define with DACs a reprocessing plan in 2 stepsFrom 2000-2009 : Convert the ANDRO trajectory files reprocessed from ARGOS messages by IFREMER into the new trajectory files =>January 2013From 2010-now: reprocessing done at DACsDAC cookbook is being compiled with detailed instructions on how to fill the more complicated variables like the cycle timing variables for each float type => ADMT13
22
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
New cycle timing variables
flags:0 : estimated from metadata1 : estimated2 : transmitted by float3 : determined by positioning system9 : date not known
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
ANDRO Velocity ATLAS (2000-2009)
=>Talk Friday M Ollitrault/ Ifremer- JP Rannou/Altran
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Format Improvement
Standardization of the tech file (DONE) Tech files contains information that are important to monitor a fleet of float behavior Nearly 700 different technical information are reported to shore by the different versions of Argo Floats
=> definition a common vocabulary for technical informationRegeneration of the tech files by the DACS
A monitoring is in place done by Ann Gronell at CSIRO Most of the anomaly detected at last audit were minor and most of them have been cleaned up
Standardization of MetaData files (Work in progress)Define a common reference tables to record float metadata such as platform, sensor,.. (Define mandatory metadata that should be filled by all DACSAdd the capability to record the change done by two way transmission means Set up a monitoring at AIC level
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Preparing the Argo Evolution
Addition of new sensors on 6% of the floats: Oxygen, Chlorophyll, Nitrate
Define how to store these new parameters : Agreement on the units , what information we keep , how we calculate the parameters
Define Real Time QC procedures for these new parameters Define Delayed Mode QC procedures for these new parameters
BIO-ARGO workshop Next November prior to ADMT
2way communication used on 15 % of the floats Data Format must evolve to handle mission changes (done)
Sampling on the vertical can be different for the different sensors : definition of multi-axis Argo netcdf Argo format NO IMPACT ON THE CORE MISSION : Old well written software work on new format
More technical parameters are sent to shore => enhance vocabularies to store these dataReduction of the time at Surface :
+ Gain in energy and safer for the floats that stay less at surface− Impact on velocity calculation as very few displacements at Surface
26
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
Lessons Learned
Open data policy is an asset for an international observing program
It's possible to process data in RT in a coherent way but need important coordination
The DAC /GDAC architecture works and is reliable
Delayed Mode processing in a coherent and subjective-less manner is challenging and requires a lot of coordination, training and cooperation
Argo Data Management system is still working on improving monitoring data Quality between Real-time and Delayed Mode processing
Reprocessing at DACS/GDACS is expensive as the ARGO dataset is becoming huge ( 1 000 000 profiles before the end of 2012)
Feedback from users is welcomed and necessary to improve the system ( [email protected] )
27
Argo
4th
Sci
entif
icw
orks
hop
Veni
ceSe
ptem
ber
2012
28
Argo data management reference documents
Reference documents http://www.argodatamgt.org/Data management web page Beginner's guide to Argo dataData management policy GDACs organization User’s manual; data formatsQuality control manual
Data accessThe FTP addresses are :
ftp://ftp.ifremer.fr/ifremer/argoftp://www.usgodae.org/pub/outgoing/argo
Data discovery : web interfacehttp://www.usgodae.org/cgi-bin/argo_select.plhttp://www.argodatamgt.org/Access-to-data/Argo-data-selection http://www.argodatamgt.org/Access-to-data/Description-of-all-floats
User Desk : [email protected]
