1

Efficient Reduction and Compression of Weather Radar Data in Universal Format

W. David PanDept. of Electrical & Computer Engineering, Univ. of Alabama in Huntsville

ONR Summer Faculty Research Fellow, Naval Research Laboratory, Monterey, CA

Paul R. HarastiVisiting Scientist Programs, University Corporation for Atmospheric Research, Boulder, CO

Michael Frost, Qingyun Zhao, John CookMarine Meteorology Division, Naval Research Laboratory, Monterey, CA

Timothy MaeseBasic Commerce and Industries, Moorestown, NJ

Lee J. WagnerAtmospheric Propagation Branch, SPAWAR Systems Center, San Diego, CA

2

Outline

• Problem

• UF files

• Challenge

• Header compression

• Radar data compression

• Software developed and test results

• Further work

3

Radar Data Assimilation System

Naval Research Lab (NRL) Monterey is developing a radar data assimilation system to enhance the safety of at-sea ship & aircraft operations by

• predicting short term changes in in-situ hazardous weather• providing decision makers with tools to exploit or mitigate those changes.

The system will• take advantage of Navy vessels with weather radar capability, e.g.,

• SPS-48E/G: HWDDC (Hazardous Weather Detection and Display Capability)

• SPY-1 Tactical Environmental Proc.

• have the capability to send radar data back to FNMOC for incorporation into the numerical weather prediction model

4

Impacts of HWDDC

– Flight safety• Reduction of weather-related aviation

mishaps

– Flight and deck operations• Avoidance of weather-related disruptions

– Ship navigation• Avoidance of storms, areas of poor visibility• Maintenance of wind over deck

5

Weather Extraction Computer

Source: Tim Maese et al., “Hazardous Weather Detection and Display Capability for US Navy Ships,” 23rd AMS Conf. on IIPS, 2007.

Weather Radar Data Files (UF Format)

5.4 MB / File x 12 Files / Hour

6

Challenge: Minimized Bandwidth Use

Ship 1

Battlegroup SIPRNet

FNMOC

3 UF files / hour / ship

Ship 2Ship n …

7

Radar Data Files in Universal Format

File Size File Name

• UF files generated approx. once every 5 mins• File size is 5.4 Mega Bytes / file• File consists of records• Each record consists of headers and data of different types• Data set available (252 UF files collected on Feb. 22, 2006)

8

Challenge

UF File Size

1 MB

The smaller, the better !

Gzip

5.4 MB 1.7

Bzip2

2.4 1.3

PAQ

Note: PAQ required 14 minutes per UF file – unacceptable for near-real-time applications

Goal:

Can we compress each UF file down to < 1 MB ?

9

Our Strategies• Divide and Conquer

– Significant redundancies in headers• Headers of neighboring records change very slightly

– Headers interleaved with data in original UF files• Render the UF files hard to compress

– Separating headers from data• Reorganizing the UF file make it easier to compress

• Goal– Large compression on headers– Lossless recovery of headers guaranteed– Data reduction w/o impacting COAMPS (Coupled

Ocean/Atmosphere Mesoscale Prediction System)– Near real-time (low delay, small memory, etc.)

10

Headers and Radar Data

Elevation

Range

Azimuth ...

Ray

Ray

.

.

.

MHB LUHB DH FH(DZ) Data

(DZ)

45 9 3+2*5 19 19

FH(SN) Data

(SN) FH(VE) Data

(VE)

21 21

FH(SW) Data

(SW) 21

FH(VV) Data

(VV)

Data stored in a record:

Length:

Headers:

MHB: Mandatory Header BlockLUHB: Local Use Header BlockDH: Data HeaderFH: Field Header________________________

Data Fields:

DZ: Reflectivity (dBZ)SN: Signal-to-Noise Ratio (dB)VE: Mean Radial Velocity (m/s)SW: Spectrum Width (m/s)VV: Valid Velocity Flags

11

Breakdown

Header

Data

12

Mandatory Header Blocks uf: [2x1 char] recordLength: 941 nonMandPosition: 46 localPosition: 46 dataPosition: 55 recordFile: 0 volumeScan: 1 rayNumber: 0 recordInRay: 1 sweepNumber: 0 radarName: [8x1 char] shipName: [8x1 char] latDegrees: 20 latMinutes: 46 latSeconds: -27776 lonDegrees: -154 lonMinutes: -57 lonSeconds: 5312 antennaHeight: 40 year: 6 month: 2 day: 22 hour: 15 minute: 42 second: 50 timeZone: [2x1 char] azimuth: 0 elevation: 12 sweepMode: 1 fxa: 0 sweepRate: 5760 genYear: 2006 genMonth: 2 genDay: 22 genShip: [8x1 char] missingFlag: 1

uf: [2x1 char] recordLength: 941 nonMandPosition: 46 localPosition: 46 dataPosition: 55 recordFile: 1 volumeScan: 1 rayNumber: 1 recordInRay: 1 sweepNumber: 0 radarName: [8x1 char] shipName: [8x1 char] latDegrees: 20 latMinutes: 46 latSeconds: -27776 lonDegrees: -154 lonMinutes: -57 lonSeconds: 5312 antennaHeight: 40 year: 6 month: 2 day: 22 hour: 15 minute: 42 second: 50 timeZone: [2x1 char] azimuth: 64 elevation: 12 sweepMode: 1 fxa: 64 sweepRate: 5760 genYear: 2006 genMonth: 2 genDay: 22 genShip: [8x1 char] missingFlag: 1

Record 0 Record 1

different entry

different entry

different entry

different entry

13

Macro Headers

MHB LUHB DH FH(DZ) Data

(DZ) FH(SN) Data

(SN) FH(VE) Data

(VE) FH(SW) Data

(SW) FH(VV) Data

(VV)

MHB LUHB DH FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV)

Record n: merge the headers (zero padding for empty headers)

MHB LUHB DH FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV)

MHB LUHB DH FH(DZ) FH(SN)

MHB LUHB DH FH(DZ) FH(SN) 00…………………0

.

.

.

Recordindex

00…………………0

168 words = 336 bytes

14

Direct Compression

0

The macro header can be compressed to about 8 KB

15

Differential Operation

MHB LUHB DH FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV)

MHB LUHB DH FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV)

Record n

Record n+1

Sparse vector with most of the entries being 0

Bit-wise Exclusive OR

Diff. Record n+1

xy 0 1

0

1

yx

yxyx

if1

if0

0 1

01

Bit-wise XOR

yyxx )(

Truth Table Properties

(Recovery)

x

yyx

• XOR operations run much faster than subtractions on computers• XOR is safe – no worry about exceeding dynamic range due to subtraction

16

Lossless Reconstruction

MHB LUHB DH FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV)

MHB LUHB DH FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV)

Record n+1

Record n

Sparse vector with most of the entries being 0Diff. Record n+1

H0

H1

H2

.

.

.

x

yx

y

=

.

.

.

=

.

.

.

H0

H1

H2

D0 = H0

D1 = H0

D2 = H1

H1

H2

Macro Header Differential Macro Header Perfectly Recon. Macro Header

Sparse and thuseasy to compress

17

Differential Macro Header

The DMH can be compressed to about 2.5 KB-- about 3 times more compression than on MH

Sea of 0’s

18

Compressed Header Sizes

Over 600:1 compression!

(from 1,677 KB)

< 3 KB

Time

19

Data Reduction• Can now focus on data compression

– Headers can be squeezed to negligibly small sizes

• Blank out data if values are below certain thresholds, as determined by the QC (quality control) requirements– DZ <= 5 dBZ, or– SN <= 10 dB

• Some data types (e.g., SN) in the original UF files are for used for QC only– Drop these data will further reduce the bandwidth load– Transmit only three types of data (DZ, SW, VE)

Data70%

Header

20

UF Compression Software Package

UF File

Header Extractor

Data Extractor

XORMH

Thresholding

BZIP2compression

Channel

BZIP2de-compression

‘uf.dat’

‘uf.dat.bz2’

DMH

‘uf.dat’

1

XOR-1

UF Reader

Data

MH UF

Writer

Recon.UFFile

DMH

‘ufzip’: Encoder

‘ufunzip’: Decoder

21

No Thresholding

Avg. Size of Compressed File (DZ, SW, and VE) = 0.4 MB

22

Thresholding

< 0.25 MB/file

Sizes of CompressedUF Files

23

OriginalDZData

ReconnoTH

ReconwithTH

UF File 1 UF File 70 UF File 190

24

Linux Screenshots

Took 3 sec to compress an UF File

Compressed file size (in bytes)

Took only 1 sec to reconstruct the UF File

Decoder

Encoder

25

Summary

UF File Size

1 MB

Gzip

5.4 MB 1.7

Bzip2

2.4 1.3

PAQ

0.12 MB

Threshold + only DZ,VE,SW transmitted

(Effective Reduction > 40: 1)

0.5 MB

Estimated upper bound on compressed UF file sizes-- more than meet the goal!

26

Future Work

Further Work• The developed UF file compression software

package will be tested for approval for ship installation by the Integrated Test Facility laboratory, Space and Naval Warfare Systems Center (SPAWAR), San Diego, CA.

Acknowledgment• ONR Summer Faculty Research Fellowship