Multithreaded ingestion of BUFR messages from the IDD John Caron Oct 8, 2008.
Introduction to BUFR
-
Upload
shea-rowland -
Category
Documents
-
view
36 -
download
1
description
Transcript of Introduction to BUFR
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Introduction to BUFRSimon Elliott
EUMETSAT
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
What is BUFR?
• Binary Universal Form for the Representation
of Meteorological Data• Used for data that are not on a regular grid, such as
observations• Conceptually equivalent to CREX, but format is
binary rather than alphanumeric
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
What does a BUFR message look like? 01000010010101010100011001010010000000000000000000110100000000110000000000000000
00010010000000000000000000111000000000000000000000000000000000000000100100000001
00000001000001000001110100001100000000000000000000000000000000000000111000000000
00000000000000011000000000000001000000010000000100000010000011000000010000000000
00000000000000000000100000000000100100001111010111011100010000000011011100110111
0011011100110111
(In other words, just an apparently random string of 0’s and 1’s!)
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Sections of a BUFR message
• 0 Indicator section
• 1 Identification section
• 2 Optional local use section
• 3 Data description section
• 4 Data section
• 5 End of message
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 0 – Indicator section
• The character string “BUFR” indicating the start of the message
• The total length of the message
• The BUFR edition number
This section contains:
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 0 - Details
• Length always 8
• Octets 1-4 “BUFR” (in CCITT IA5)
• Octets 5-7 Total length of message
(including Section 0)
• Octet 8 Edition number
(currently 4, but 3 is still used)
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Now, let’s go back and look at that BUFR message again…
‘B’ ‘U’ ‘F’ ‘R’ end of section 0 +
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
binary string 01000010010101010100011001010010000000000000000000110100000000110000000000000000
00010010000000000000000000111000000000000000000000000000000000000000100100000001
00000001000001000001110100001100000000000000000000000000000000000000111000000000
00000000000000011000000000000001000000010000000100000010000011000000010000000000
00000000000000000000100000000000100100001111010111011100010000000011011100110111
0011011100110111
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 1 – Identification section
• The table versions referred to by this message• An overall description of the message contents,
including:– The originating centre and sub-centre
– The data category and sub-category
– A representative date and time
• Whether or not the optional section is included
This section contains:
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 1 – Details(example based upon BUFR edition 3)
• Length at least 18
• Octets 1-3 Length of section
• Octet 4 Master table (0 for WMO, 10 for IOC, etc.)
• Octet 5-6 Originating sub-centre and centre
• Octet 7 Update sequence number
• Octet 8 Flag (Optional section?)
• Octets 9-10 Data category and sub-category
• Octets 11-12 Master and local table version numbers
• Octets 13-17 Date and time typical of message contents
• Octets 18-?? Reserved for local use
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Now, let’s go back and look at that BUFR message again…
‘B’ ‘U’ ‘F’ ‘R’ end of section 0 +
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 1 | 2 |
binary string 01000010010101010100011001010010000000000000000000110100000000110000000000000000
octet number 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
binary string 00010010000000000000000000111000000000000000000000000000000000000000100100000001
end of section 1 +
octet number 13 | 14 | 15 | 16 | 17 | 18 |
binary string 00000001000001000001110100001100000000000000000000000000000000000000111000000000
00000000000000011000000000000001000000010000000100000010000011000000010000000000
00000000000000000000100000000000100100001111010111011100010000000011011100110111
0011011100110111
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 2 – Optional section
• It typically contains additional information of use to the ADP centre, such as– Database keys to aid searching for specific data
without decoding the message– Anything else a processing centre may find useful
This section is defined by the ADP (Automated Data Processing) centre generating or using the message
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 3 – Data description section
• A count of the number of data subsets (typically individual observations)
• Flags indicating whether or not the data are compressed or uncompressed and observed or forecast
• A list of the data elements (fields) that are contained in each data subset
This section contains:
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 3 - Details
• Length at least 10• Octets 1-3 Length of section• Octet 4 Set to zero• Octets 5-6 Number of subsets• Octet 7 Flag (Obs?, Compressed?)• Octets 8-?? List of descriptors• Each descriptor 2 bits F, 6 bits X, 8 bits Y
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Now, let’s go back and look at that BUFR message again…
‘B’ ‘U’ ‘F’ ‘R’ end of section 0 +
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 1 | 2 |
binary string 01000010010101010100011001010010000000000000000000110100000000110000000000000000
octet number 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
binary string 00010010000000000000000000111000000000000000000000000000000000000000100100000001
end of section 1 +
octet number 13 | 14 | 15 | 16 | 17 | 18 | 1 | 2 | 3 | 4 |
binary string 00000001000001000001110100001100000000000000000000000000000000000000111000000000
end of section 3 +
octet number 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 |
binary string 00000000000000011000000000000001000000010000000100000010000011000000010000000000
00000000000000000000100000000000100100001111010111011100010000000011011100110111
0011011100110111
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 4 – Data section
• The actual data as specified by Section 3
• One of two formats is used– Compressed– Uncompressed
• Such data are still packed, but not as efficiently as compressed data usually are
This section contains:
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 4 - Details
• Octets 1-3 Length of section
• Octet 4 Set to zero
• Octets 5-?? Binary data as specified
by Section 3
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Now, let’s go back and look at that BUFR message again…
‘B’ ‘U’ ‘F’ ‘R’ end of section 0 +
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 1 | 2 |
binary string 01000010010101010100011001010010000000000000000000110100000000110000000000000000
octet number 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
binary string 00010010000000000000000000111000000000000000000000000000000000000000100100000001
end of section 1 +
octet number 13 | 14 | 15 | 16 | 17 | 18 | 1 | 2 | 3 | 4 |
binary string 00000001000001000001110100001100000000000000000000000000000000000000111000000000
end of section 3 +
octet number 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 |
binary string 00000000000000011000000000000001000000010000000100000010000011000000010000000000
end of section 4 +
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
binary string 00000000000000000000100000000000100100001111010111011100010000000011011100110111
0011011100110111
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 5 – End section
• The character string “7777” indicating the end of the message
• Checking for this indicator can be useful to detect some types of data corruption (especially missing bytes in the rest of the message) since the total length of the message is known from Section 0
This section contains:
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Now, let’s go back and look at that BUFR message one last time!
‘B’ ‘U’ ‘F’ ‘R’ end of section 0 +
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 1 | 2 |
binary string 01000010010101010100011001010010000000000000000000110100000000110000000000000000
octet number 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
binary string 00010010000000000000000000111000000000000000000000000000000000000000100100000001
end of section 1 +
octet number 13 | 14 | 15 | 16 | 17 | 18 | 1 | 2 | 3 | 4 |
binary string 00000001000001000001110100001100000000000000000000000000000000000000111000000000
end of section 3 +
octet number 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 |
binary string 00000000000000011000000000000001000000010000000100000010000011000000010000000000
end of section 4 + ‘7’ ‘7’
octet number 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 1 | 2 |
binary string 00000000000000000000100000000000100100001111010111011100010000000011011100110111
‘7’ ‘7’ + end of section 5
octet number 3 | 4 |
binary string 0011011100110111
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
BUFR Descriptors
• Section 3 contains a list of BUFR descriptors• These describe the data elements that are
contained in Section 4• Most descriptors are references to BUFR Tables
B, C and D• Using the list of descriptors in Section 3, together
with the tables, it is possible to unpack the data in Section 4
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Types of BUFR descriptors
• Element descriptors (Table B)• Replication descriptors• Operator descriptors (Table C)• Sequence descriptors (Table D)• Specified by 3 numbers in 16 bits (2 octets)
– F: 2 bits 0-3
– X: 6 bits 0-63
– Y: 8 bits 0-255
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Element descriptors
• Defined by entries in Table B• F is 0• Each element descriptor describes an encoded
value, such as:– The value of a meteorological parameter (e.g. mean sea
level pressure, temperature, wind speed)– Instrument details– Location or date and time information– Quality control information
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Replication Descriptors
• Describe the repetition of one or more element, operator, sequence or other replication descriptors
• Used for repetitive data such as the individual levels in vertical soundings or temperature profiles
• Can be:– Fixed - the number of repetitions is pre-determined and
the same for all data subsets– Variable - the number of repetitions can differ from one
subset to the next (ie. delayed replication)
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Replication descriptors - continued• Replication descriptors are defined by three numbers
F X Y• F is 1• X is an integer between 1 and 63
– Defines the number of descriptors to be repeated
• Y is an integer between 0 and 255– Defines how many times the X descriptors are to be
repeated– A count of zero indicates delayed replication, where the
repeat count is stored in the data section and can change from one data subset to another.
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Operator descriptors• Defined by entries in Table C• F is 2• Describe changes to be made to other descriptors• Operators exist for applications such as:
– Changing the scaling and packing of data– Adding quality control or other associated fields– Describing the descriptors to which quality control
information applies– Substituting a better value for an element, while
retaining the original value
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Sequence descriptors
• Defined by entries in Table D• F is 3• Shorthand notations for pre-defined lists of other
element, replication, sequence and operator descriptors
• Not really necessary, but useful in reducing the overhead involved in transmitting data in BUFR:– Replace a commonly-used sequence of descriptors with
a single descriptor, and thereby reduce the overall length of Section 3
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
BUFR tables
• Table A– Data Categories, used in Section 1
• Table B– Element descriptors, used in Section 3
• Table C– Operator descriptors, used in Section 3
• Table D– Sequence descriptors, used in Section 3
• Code and Flag tables– Numerical values to be encoded where the element values are
qualitative, used in Section 4
There are many different tables involved in BUFR:
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table A• Defines the general category of the data contained
in the BUFR message• Encoded in Section 1• Examples of typical entries:
Code figure Meaning 0 Surface data – land 1 Surface data – sea 2 Vertical soundings (non-satellite) 3 Vertical soundings (satellite) … … 6 Radar data … … 10 Radiological data 12 Surface data (satellite) … … 31 Oceanographic data
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B• Describes the individual values that are encoded• Element descriptors are grouped according to
classes (i.e. X value)
Class Number Class Name Class Number Class Name
01 Identification 12 Temperature
02 Instrumentation 13 Hydrological
… … 14 Radiation and radiance
04 Location (time) … …
05 Location (horizontal-1) 19 Synoptic features
06 Location (horizontal-2) 20 Observed phenomena
07 Location (vertical) 21 Radar data
… … … …
11 Wind and turbulence 33 Quality information
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Class 01 – Identification(excerpt)
TABLE
REFERENCE
F X Y
TABLE
ELEMENT
NAME
UNIT SCALEREFERENCE
VALUE
DATA
WIDTH
(BITS)
0 01 001 WMO block number Numeric 0 0 7
0 01 002 WMO station number Numeric 0 0 10
0 01 003 WMO region number Code table 0 0 3
0 01 005 Buoy/platform identifier
Numeric 0 0 17
0 01 006 Aircraft flight number
CCITT IA5 0 0 64
0 01 007 Satellite identifier Code table 0 0 10
0 01 011 Ship or mobile land station identifier
CCITT IA5 0 0 72
0 01 015 Station or site name CCITT IA5 0 0 160
0 01 063 ICAO location indicator
CCITT IA5 0 0 64
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Class 11 – Wind and turbulence(excerpt)
TABLE
REFERENCE
F X Y
TABLE
ELEMENT
NAME
UNIT SCALEREFERENCE
VALUE
DATA
WIDTH
(BITS)
0 11 001 Wind direction Degree true 0 0 9
0 11 002 Wind speed m s-1 1 0 12
0 11 003 U-component m s-1 1 -4096 13
0 11 004 V-component m s-1 1 -4096 13
0 11 021 Relative vorticity s-1 9 -65536 17
0 11 031 Degree of turbulence Code table 0 0 4
0 11 032 Height of base of turbulence
m -1 -40 16
0 11 033 Height of top of turbulence
m -1 -40 16
0 11 034 Vertical gust velocity m s-1 1 -1024 11
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B• Columns are:
– Table reference– Element name– Unit– Scale– Reference value– Data width (in bits)
• Scale, reference value, and bit width are chosen so that the desired range of possible data values can be stored in BUFR as positive integers– Preserves the machine-independence of BUFR
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B reference• Expressed as 3 small numbers F X Y• Used to refer to this descriptor• F is always 0 for an element descriptor• X is in the range 0 to 63 and refers to a broad
class of elements– Classes 48 to 63 are reserved for local use
• Y is in the range 0 to 255 and refers to the individual descriptor in the class– Within all classes, descriptors 192 to 255 are reserved
for local use
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B element name
• Natural language description of the meaning of the value
• Usually English but could be translated to other languages
• For example:– Brightness temperature– Total precipitation past 3 hours– Wind speed at 10m
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B unit• The units used for the value
– Normally SI units are used– “CCITT IA5” (the international version of ASCII) is used
for character data such as identifiers– “Code Table” is used for qualitative data where only one of
a set of possible values can be applicable in a given data subset
– “Flag Table” is used for qualitative data where more than one of a set of possible values may be applicable in a given data subset
– For qualitative data, the coded values are references to the Code and Flag tables
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B scale
• Scale– Power of 10 by which to multiply the data value before
packing
– Determines the precision with which the data are encoded
– A scale of 2 means 2 decimal places of precision (eg. 273.16)
– A scale of –1 means that the data values are rounded to the nearest multiple of 10
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B reference value• Used to subtract an offset where negative data
have to be encoded
• Table B contains the value (multiplied by the scale) of the offset to be subtracted
• For example, scale=2, reference value -9000 means that -90.00 is to be subtracted before scaling (ie. -9000 after scaling), allowing values as negative as -90.00 to be represented
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B data width
• The number of bits to be used to encode the value• If all bits are set to ones when encoding (ie. a
value of (2n-1) when n is the data width), then this denotes a “missing” value.
• If the scale is s, the reference value is r, and the data width is n, then the representable range of values is:– Minimum (10-s r)– Maximum (10-s (2n-2+r))
and (10-s (2n-1+r)) denotes the “missing” value.
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B examples
Table reference
F X Y
Element name Unit Scale Reference value
Data width
0 11 012 Wind speed at 10m m s-1 1 0 12
0 13 020 Total precipitation past 3 hours kg m-2 1 -1 14
0 20 003 Present weather Code table 0 0 9
0 08 001 Vertical sounding significance Flag table 0 0 7
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B examples - continued• 0 11 012 - Wind speed at 10m• Scale=1, Reference value=0, Data width=12• Precision is one decimal place (ie. 0.1 m s-1)• Minimum representable value is:
(10-1×0) = 0.0 m s-1
• Maximum representable value is:
(10-1×(212-2+0)) = 409.4 m s-1
• “Missing” value is:
(10-1×(212-1+0)) = 409.5 m s-1
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B examples - continued• 0 13 020 - Total precipitation past 3 hours• Scale=1, Reference value=-1, Data width=14• Precision is one decimal place (ie. 0.1 kg m-2)• For this descriptor, -0.1 kg m-2 is a special value for trace,
according to a specific note in Table B• Minimum representable value is:
(10-1×-1) = -0.1 kg m-2 (= trace)• Maximum representable value is:
(10-1×(214-2-1)) = 1638.1 kg m-2
• “Missing” value is:(10-1×(214-1-1)) = 1638.2 kg m-2
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table B examples - continued• 0 20 003 - Present weather• Scale=0, Reference value=0, Data width=9• Coded values are integers since Scale=0• Minimum representable value is:
(100×0) = 0• Maximum representable value is:
(100×(29-2+0)) = 510• “Missing” value is:
(100×(29-1+0)) = 511• One must refer to Code Table 0 20 003 in order to
discover the actual meaning of each coded value
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
0 20 003 – Present WeatherCode Table (excerpted)
Code figure Meaning
0 Cloud development not observed or not observable
1 Clouds generally dissolving or becoming less developed
…
10 Mist
11 Patches of shallow fog or ice fog
13 Lightning visible, but no thunder heard
…
171 Snow, slight
172 Snow, moderate
173 Snow, heavy
…
511 Missing
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Code tables vs. Flag tables(choice of one vs. choice of more than one)
0-01-003 0-08-001
WMO region number Vertical sounding significanceCode figure Meaning Bit number Meaning
0 Antarctica 1 Surface
1 Region I 2 Standard level
2 Region II 3 Tropopause level
3 Region III 4 Maximum wind level
4 Region IV 5 Significant
5 Region V temperature level
6 Region VI 6 Significant wind level
7 Missing value All 7 Missing value
For a Code table, the value stored in Section 4 is the code figure corresponding to the applicable meaning.
For a Flag table of N bits, the value stored in Section 4 is (2(N-bit#) + 2(N-bit#) + …) for the bit(s) corresponding to each applicable meaning. An extra bit is always present in order to be able to distinguish “all meanings applicable” from “missing”.
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Some other important regulations pertaining to Table B
• Elements in classes 01 – 09 are “coordinate” descriptors which
remain in effect until redefined or until the end of the subsetException: when two identical descriptors from classes 04 – 07 are listed consecutively, they define the boundaries of a range
• Similar descriptors exist in “coordinate” vs. “non-coordinate” classesExample: 0 07 004 and 0 10 004 are both “Pressure” with identical scale, reference value and bit width; however, the former is a “coordinate” for use when pressure is the main defining coordinate measured in the vertical direction (e.g. in radiosondes) vs. the latter which is a “non-coordinate” for use when pressure is a derived value (e.g. an aircraft calculating pressure as a function of an observed or measured height)
• Class 08 contains significance qualifiers which can be used to report
qualitative information and which can be explicitly “cancelled”Example: 0 08 011 with value 12 can indicate that we are talking about a “cloud”
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table C• Describes the various operators• Columns are:
– Table reference F X• F is 2• X is an integer between 0 and 63• There is no sub-range of X values reserved for local use
– Operand• A number between 0 and 255
– Operator name• A short name describing the operation
– Operator definition• A detailed description of the operation and its effects
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table CTABLE
REFERENCE
F X
OPERAND OPERATOR NAME OPERATOR DEFINITION
2 01 Y Change data width Add (Y-128) bits to the data width given for each data element in Table B, other than CCITT IA5, code or flag tables
2 02 Y Change scale Add (Y-128) to scale in Table B for elements which are not CCITT IA5, code or flag tables
2 03 Y Change reference value Subsequent element descriptors define new reference values for corresponding Table B entries. Each new reference value is represented by Y bits in Section 4…
2 04 Y Add associated field Precede each element with Y bits of information (e.g. quality marker).
2 05 Y Signify character Y characters from CCITT IA5 are inserted as a field of (Y*8) bits in length.
2 06 Y Signify data width for following local descriptor
Y bits of data are described by the immediately following local descriptor from Table B
• This is just an excerpt – there are many other (even more complicated!) operators in Table C.• There are also many important notes to Table C describing, e.g. how to cancel an operator.
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table C example• Table reference F=2 X=01• Operand, in this case represented as Y• Operator name “Change data width”• Operator description:
“Add Y-128 bits to the data width given for each data element in Table B, other than CCITT IA5 (character) data, code or flag tables”
• According to a note under Table C, this operator is cancelled (ie. effect is turned off) by repeating the operator with Y=0, or at the end of each data subset
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table C example - continuedThe “Change data width” operator causes the data width to be changed for subsequent elements, in effect giving them a larger (or smaller) range than is otherwise prescribed within Table B. Thus, it can be used to:
• encode values that exceed the usual representable range for a descriptor, instead of having to introduce a new Table B descriptor (note: in such cases, Y > 128)• reduce the size of the data (and thus the overall encoded message as well!) if the required data range can be encoded using a smaller data width than provided within Table B (note: in such cases, Y < 128)
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table C example -continued
As an example, one of the standard descriptors for the height coordinate of an observation is 0 07 007 with unit=m, scale=0, reference=-1000, data width=17, giving a representable range of –1000 m to 130070 m. If one needed to encode a value larger than this, then the 2 01 operator could be used to increase the data width. For example, use of the operator 2 01 130 before the 0 07 007 descriptor would increase its data width to 19 bits and therefore allow values up to 523286 m.
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table D• Describes sequences of descriptors• Columns are:
– Table reference F X Y• F is 3• X is in the range of 0 to 63 and refers to a broad category of sequences
– Categories 48 to 63 are reserved for local use
• Y is in the range of 0 to 255 and refers to the individual sequence in the category
– Within all categories, entries 192 to 255 are reserved for local use
– Table references• List of other descriptors, including element descriptors (Table B),
replication descriptors, operator descriptors (Table C) and other sequence descriptors (Table D)
– Element name• Not really necessary, but makes reading the tables easier
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table D categories
• Categories correspond to the X value of the underlying sequence descriptor.
Category number Category name
01 Location and identification sequences
02 Sequences common to surface data
03 Sequences common to vertical soundings data
04 Sequences common to satellite observations
05 Sequences common to hydrological observations
06 Sequences common to oceanographic observations
07 Surface report sequences (land)
08 Surface report sequences (sea)
… …
18 Radiological report sequences
21 Radar report sequences
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Category 01 – Location and Identification Sequences (excerpt)TABLE REFERENCE TABLE REFERENCES ELEMENT NAME
3 01 001 0 01 001
0 01 002
WMO block number
WMO station number
3 01 011 0 04 001
0 04 002
0 04 003
Year
Month
Day
3 01 012 0 04 004
0 04 005
Hour
Minute
3 01 024 0 05 002
0 06 002
0 07 001
Latitude (coarse accuracy)
Longitude (coarse accuracy)
Height of station
3 01 038 3 01 001
0 02 011
0 02 012
3 01 011
3 01 012
3 01 024
(Land station for vertical soundings)
WMO block and station number
Radiosonde type
Radiosonde computational method
Date
Time
Lat/long (coarse accuracy), height of station
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table D example
Table reference F X Y
Table References
Element name
(Buoy/platform – fixed)
3 01 033 0 01 005 Buoy/platform identifier
0 02 001 Type of station
3 01 011 Date
3 01 012 Time
3 01 021 Latitude and longitude (high accuracy)
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Table D example - continued• In this example, the sequence consists of five
descriptors• Two of these are Table B element descriptors• The other three are Table D sequence descriptors,
which in turn represent other sequences of descriptors
• Obviously the repeated replacement of Table D descriptors with lists of descriptors must not continue forever, and a Table D descriptor cannot include itself, directly or indirectly
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Replication descriptors - example• The 4 descriptors:
– 1 03 002 Repeat three descriptors twice– 0 22 003 Direction of swell waves– 0 22 013 Period of swell waves– 0 22 023 Height of swell waves
• Are equivalent to the 6 descriptors:– 0 22 003 Direction of swell waves– 0 22 013 Period of swell waves– 0 22 023 Height of swell waves– 0 22 003 Direction of swell waves– 0 22 013 Period of swell waves– 0 22 023 Height of swell waves
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Delayed replication descriptor - example
• The 3 descriptors:– 1 01 000 Delayed replication of one descriptor– 0 31 001 Replication factor (8 bit)– 3 03 012 Winds at pressure levels
• Are equivalent to:– 3 03 012 Winds at pressure levels– 3 03 012 Winds at pressure levels– … for as many times as specified by the 0 31 001 replication factor
• The number of repeats for each data subset is stored in the data. Each data subset has its own repetition count.
• In addition to 8-bit delayed replication, there is also 1-bit and 16-bit delayed replication
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
A caveat…The previous example is worth a closer look. Specifically, when replicating a sequence, the replication takes place before the sequence expansion!
For example, note that:
1 01 004
3 03 012 ( = 0 07 004, 0 08 001, 0 11 001, 0 11 002 )
is equivalent to: and not to:
3 03 012 0 07 004
3 03 012 0 07 004
3 03 012 0 07 004
3 03 012 0 07 004
0 08 001
0 11 001
0 11 002
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 4 – Detailed description• The form of the data in Section 4 varies
depending upon the descriptors and flags in Section 3– One of the main differences concerns whether
the data are uncompressed or compressed– Delayed replication and compression may be
combined, but only if the number of repetitions is the same for each subset within a particular BUFR message
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 4 – Uncompressed data• Each data item takes up the number of bits
specified by the data width in Table B, as adjusted by operators or as otherwise specified.
• The order for N data subsets, each with M data values is:
Set 1 Value 1, Set 1 Value 2, Set 1 Value 3, … Set 1 Value M
Set 2 Value 1, Set 2 Value 2, Set 2 Value 3, … Set 2 Value M
…
Set N Value 1, Set N Value 2, Set N Value 3, … Set N Value M
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Section 4 – Compressed data• Each data item is encoded by its minimum over all
the data subsets (using the data width specified in Table B or otherwise), the number of bits used to encode the increments (using 6 bits), and a list of increments to be added to the minimum.
• The order for N data subsets, each with M data values is:
Min 1, Nbits 1, Set 1 Inc 1, Set 2 Inc 1, … Set N Inc 1Min 2, Nbits 2, Set 1 Inc 2, Set 2 Inc 2, … Set N Inc 2….Min M, Nbits M, Set 1 Inc M, Set 2 Inc M, … Set N Inc M
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Summary• BUFR is flexible
– It can represent a wide range of data types
• BUFR is table-driven– Tables A, B, C and D (and code and flag tables) contain
majority of information needed to encode and decode data
• BUFR is self-describing– The data description section describes the data, using
element (Table B), replication, operator (Table C) and sequence (Table D) descriptors
• Replication descriptors and sequence descriptors can be used to reduce the size of the data description
• Compression can be used to reduce the size of the data themselves
WORLD METEOROLOGICAL ORGANIZATION
RA-VI Regional Training on BUFR and Migration to Table Driven Code FormsLangen, Germany, 17 - 20 April, 2007
Acknowledgements
Based on:• WMO No. 306 Manual on Codes, Volume I.2
(http://www.wmo.int/web/www/WMOCodes/ManualCodes/WMO306vol-I-2PartB.pdf)
• Guide to WMO Table-Driven Code Forms FM94 BUFR and FM95 CREX (http://www.wmo.int/web/www/WMOCodes/Guides/BUFRCREXPreface_en.html)
Special thanks to:• Jeff Ator NOAA, United States of America• Charles Sanders BOM, Australia (retd.)• Joël Martellet WMO