Observation Screening

17 June, 2006 3D-VAR/ODB Training, Budapest

Observation Screening

Kertész Sándor

3D-VAR/ODB TrainingBudapest, 7 June, 2006


OUTLINE

• ODB primer• Observation data flow• Screening QC and decisions in ODB• Screening inputs• How is screening working?• Output listings• Exercises


ODB primer

• ODB is Observational Database

• Basic data structure is called table. Tables is made up of data columns

• Set of tables (with relations) forms a database type. E.g. ECMA that contains all the observations to be used in a DA system

• Information on tables, columns etc.•http://www.cnrm.meteo.fr/gmapdoc/meshtml/DOC_odb/odb.html

•In „Doc” directory on regatta: Assimilation_Chapter9_v2.3.pdf

•ODB training!


ODB primer

• An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!)

• ODB data is divided into pools for parallel runs (to ensure load balance)

• To look into ODB you need to run the viewer with your view!

• The view define the retrieval!


ODB primer

• View definition (e.g. to retrieve AIREP data)

CREATE VIEW myview AS

SELECT

obstype, lat, lon, varno, obsvalue, status.blacklist@body

FROM hdr, body

WHERE obstype == 2

View name

Retrieved columns

Selection criteria

Involved tables


ODB primer

• The concept of sub-bases was invented to have the choice for handling observation types separately. E.g.: ECMA.conv, ECMA.amsua, ECMA.amsub etc…

• These ODB sub-bases must be merged into a new ODB

• Many applications are working on this kind of ODB that is called virtual basis


Observation Screening

• The observation screening is the last step in the pre-processing chain to produce observation inputs for the variational analysis

• Screening performs various quality control checks on data

• ALADIN screening is very similar to the global (ARPEGE) one

• Screening uses an ECMA ODB as input

• The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow …


Obs dataflow: BATOR

• BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools

• We need: #pools = # CPUs for further applications!

• These ODB sub-bases must be merged into a new ECMA ODB using programme SHUFFLE

• Further applications are working on this ODB virtual basis


Obs dataflow: LAMFLAG

• The resulting ODB is not suitable for ALADIN screening since it is not working with observations outside the C+I zone Abort in GEFGER!

• Solution: the input ODB must be restricted to the C+I zone programme LAMFLAG

E

C+Izone@hdr=0

zone@hdr=1

LAMFLAG writes status flags into ODB


Obs dataflow: LAMFLAG namelists

&NAMFCNT

LOBSONLY=.FALSE.

/END

&NAMFGEOM

ELAT0=46.2447006399999978,

ELON0=17.0000000000000000,

ELATC=46.2447005948822678,

ELONC=16.9999999982550491,

EDELX=12176.1563281414165,

EDELY=12176.1563281414165,

NDLUN=1,

NDGUN=1,

NDLUX=229,

NDGUX=205,

Z_CANZONE=1500.,

LVAR=.T.,

LNEWGEOM=.F.,

/END

&NAMFOBS

LSYNOP=.T.,

LAIREP=.T.,

LSATOB=.T.,

LDRIBU=.F.,

LTEMP=.T.,

LPILOT=.T.,

LSATEM=.T.

LPAOB=.F.,

LSCATT=.F.,

/END

Model geometry settings

Used for screening

Observation filterIf .TRUE.: selection is based only on observation filter!


Obs dataflow: SHUFFLE reduction

• LAMFLAG puts flags into ODB but does not reduce it!

• SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB

• Then another SHUFFLE run is needed to put sub-bases together again (merging)

• This results in a virtual base ECMA ODB again that is now ready for used as screening input


Obs dataflow: Screening

• Screening puts its quality control flags and decisions into the input ECMA ODB

• QC flags, decisions for reports (in table hdr):

•status status_t

•event1 report_event1_t

•blacklist report_blacklist_t

status_t active bit1 passive bit1 rejected bit1 blacklisted bit1 ….

report_event1_t no_data bit1 all_rejected bit1 bad_practice bit1 rdb_rejected bit1 rdb_activated bit1 whitelist_activated bit1 horipos_outrange bit1 vertpos_outrange bit1 time_outrange bit1 redundant bit1 land bit1 sea bit1 …..

report_blacklist_t obstype bit1 statid bit1 codetype bit1 instype bit1 date bit1 time bit1 lat bit1 lon bit1 stalt bit1 scanpos bit1 retrtype bit1 …..

Passive and blacklisted are a subset of rejected!


Obs dataflow: Screening

• QC flags, decisions and additional infromation for observed values (in table body):

•status status_t

•event1 report_event1_t

•blacklist datum_blacklist_t

• fg_depar = obs-guess

datum_event1_t

vertco_missing bit1 obsvalue_missing bit1 fg_missing bit1 rdb_rejected bit1 rdb_activated bit1 whitelist_activated bit1 bad_practice bit1 vertpos_outrange bit1 reflevel_outrange bit1 fg2big bit1 depar2big bit1 obs_error2big bit1 datum_redundant bit1 level_redundant bit1 ….

datum_blacklist_t obstype bit1 statid bit1 codetype bit1 instype bit1 date bit1 time bit1 lat bit1 lon bit1 ….

status_t active bit1 passive bit1 rejected bit1 blacklisted bit1 ….


Obs dataflow: Compression

• After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion

• A CCMA ODB consists of only those tables and column that are needed for the analysis

• The resulting CCMA ODB after SHUFFLE contains only the active reports and observations!

• This CCMA ODB is the input of the minimization (e131)

• The observation data flow is finished!


Obs dataflow summary

BATOR

Screening

SHUFFLE (Reduc)

SHUFFLE (Merge)

LAMFLAG

SHUFFLE (ECMA->CCMA)

ECMA.sb1

ECMA.sb2 …

ECMA (Full)

SHUFFLE (Merge)

ECMA.sb1

ECMA.sb2 …

CCMA

ECMA (C+I)


How to run screening?

• It is configuration 002:

MASTER –c002 –maladin –vmeteo –t001 –aeul –eMIN1

• Inputs:

• ECMA ODB (observation, sigma-o, blacklisting)

• First guess file under 5 different names:

• ICMSHMIN1INIT, ICMSHMIN1IMIN, ICMRFMIN10000, ELSCFMIN1ALBC000, ELSCFMIN1ALBC

• Constants, statistics

• Namelists


Constants

• errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF)

• rt_coef_atovs_neepred_ieee.dat: coefficients for the RTM (received from MF), binary file

• rszcoef_fmt.dat: the same the previous file but in ASCII format

• bcor_noaa.dat: bias correction file

• chanspec_noaa: obsolete?

• rmtberr_noaa.dat: obsolete?

• cstlim_noaa.dat: obsolete?


Screening specific namelists

• These parameters should be set like this for 3D-VAR:

• NUPTRA = -1 , number of updates of the trajectory (NAMVAR)

• NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR)

• LOBS = .TRUE., real obs (NAMCT0)

• LOBSC1 = .FALSE., (NAMCT0), traj. is not computed

• LSIMOB = .FALSE., no simelated obs (NAMCT0)

• LSCREEN = .TRUE., (NAMCT0), screening is on!

• LSCRE4D = .FALSE. , 3D screening (NAMSCC)

• Tunable parameters are in NAMSCC and NAMOBS (discussed later)!


How is screening working?

DECIS

Independent decisions: performed separately for each report, RDB flag is assigned to reports

Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected

Dependent decisions: dependency on other reports or obs, or the previous decisions

rdbflag@hdr

rdbflag@body

Decisions are taken in subroutine:


Independent decisions

•Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT…

•Blacklisting (BLACKCLN, BALCKSAT)

•Background quality control (FIRST)


Blacklisting

• Blacklisting information is placed into ODB by BATOR

• In the screening a separate selection is applied for satellite data (BLACKSAT)

• Blacklisting is also performed in BLACKCLN:

• Blacklisted obs will be rejected

• Various blacklisting for SATOB (e.g. QI < 85)

• Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is .T.)

• LSLRW10: Wind 10m, LSLRT2: T 2m, LSLRRH2: THU 2m (default is .T. meaning rejection over land. Will get a passive status!)


Blacklisting

• Also in BLACKCLN:

• Height-based rejection. Controlled via NAMOBS LHDLREJ (default is .T., rejected obs will get passive status!)

• Orographic rejection limit is applied:

•LHDRW10: Wind 10m, LHDRT2: T 2m, LHDRRH2: RHU 2m (default is .T.)

• AIREP data and significant levels for TEMP and PILOT are not used below the model surface!

• RHU and Q is rejected above 300 hPa

• AIREP is not used below a certain pressure


Background QC (FIRST)

• From the BLUE theory we know:

• For a given location we expect:

RHBHxyxy Tb

Tb HHE ))(())((

2

22

1)(

b

o

b

bxHy

222))(( obbxHy

Values are read from file errgrib and interpolated to obs locations

Values are read from ODB


Background QC (FIRST)

• Flags are assigned to observations by checking:

Flaglimit

xHy

b

b

2)(

Based on predefined flaglimits 4 flag values can be assigned to obs

0: correct

1: probably correct

2: probably incorrect

3: incorrectFlaglimits are defined in DEFRUN


Dependent decisions

• Remove redundant surface levels from TEMP/PILOT (REDSL)

• Vertical consistency of profiles (VERCO)

• Removal of duplicated reports (DUPLI): two reports with the same ID, time an location …

• Redundancy check (REDUN)

• Thinning for AIREP (THIAIR)

• Specific scatterometer quality control

• Thinning of satellite data (THINN)


Redundancy check (REDUN)

• Applied for all active reports which are co-located and originate from the same station. Separate treatment for:

• Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected

• Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius

• TEMP and PILOT

• SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 hPa


Thinning of AIREP data (THIAIR)

• One flight consist of a set of reports

• Thinning is performed for each flight separately

• 3D boxes are constructed around model levels

• In each box the report closest to the analysis date with the most active observations are selected

• Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km!

• By default box size less than 50 km cannot be used in ARPEGE, code modification is needed


Thinning of satellite data (THINNER)

• For radiances a horizontal thinning is performed in two steps:

• First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD1C(sensor))

• Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD1C(sensor))

• Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, …

• Selection criterion:

• Sea is preferred over land

• Clear sky pixel is preferred over a cloudy one

• Obs closer to the analysis date is preferred


Thinning of satellite data (THINNER)

• For SATOB a the thinning is performed:

• Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km)

• Vertically around model levels

• In each box the report closest to the analysis date with the most active observations are selected

• QI values are also used in the selection


Sreening output

• Valuable summary about screening decisions can be found in the NODE file:

• Try „grep SCREENING STATISTICS” to get:

• STATUS summary

• EVENT summary

• Number of variables, departures and missing departures

• Diagnostic JO-table


Sreening output

• Summary at „grep SCREENING STATISTICS”

Report and data status summary

OB.TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED

1 725 44 0 681 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 61 60 0 1 0 6 61 59 0 2 0 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ---------------------------------------------------------------------------------------- TOT 847 163 0 684 0

OB.TYP DATA ACTIVE PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 9186 7271 1218 1915 705 6 3438 2050 64 1388 72 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ------------------------------------------------------------------------- TOT 17200 9485 3128 7715 4639


Sreening output

• Summary at „grep SCREENING STATISTICS”

Diagnostic Jo table

Obstype 1 === SYNOP, Land stations and ships

-------------------------------------------------- Codetype 11 === SYNOP Land Manual Report Variable DataCount Jo_Costfunction JO/n ObsErr BgErr U10 1380 2250.937197524 1.63 0.200E+01 0.000E+00 H2 721 834.1196058351 1.16 0.100E+00 0.000E+00 Z 567 892.5077244762 1.57 0.785E+02 0.000E+00 T2 721 2138.950482967 2.97 0.140E+01 0.000E+00 Q 721 1577.382611689 2.19 0.998E-03 0.000E+00 ---------- --------------------------- -------- ObsType 1 Total: 4110 7693.897622492 1.87


Sreening output

• Observation monitoring available at:

http://pc2088.met.hu/monitor/start.php


Thank you for your attention!

Any question?


Exercises

1. Run screening for all the prepared observations with script ~/workdir/3dvar/Assim and study the output listing! (You have to put an exit after screening in the script).


Exercises2. Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs

(except Z) are really rejected, and find the reason for it. Try this view first:

DEFINE VIEW bp AS

SELECT

obstype, status.active@hdr, varno,

obsvalue, fg_depar, press, status.active@body,

status.passive@body, status.blacklisted@body, status.rejected@body,

event1.land@body, event1.datum_redundant@body

FROM hdr, body

WHERE (obstype == 1 OR obstype == 5) && statid = ‘12843 ‘

Guidance: A prepared viewer par file and sql file can be found in ~wshop01/workdir/OdbViewer as bp.par bp.sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp.par” to run viewer.

If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of event@body!


Exercises

3. Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair.F90 to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in include.inc!

Compare the default and the new airep thinning in the output listings or in the ODB!

Observation Screening

Documents

Transcript of Observation Screening