A Neural Network PMW/IR Combined Procedure

for Short Term/Small Area

Rainfall Estimates

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Francisco J. Tapiador & Chris Kidd

University of Birmingham, UK

Vincenzo Levizzani

National Council of Research, Italy

Frank S. Marzano

University of L’Aquila, Italy

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

• Objectives of today’s presentation1. Present a methodology of data fusion of IR and PMW data at global scale:

• Short term, large coverage and high resolution rainfall estimates• Methodology to be applied to MSG (soon) and GPM products

2. Assess the quality of these estimates: • Intercomparison / Validation: HM method• Down-top approach

3. Present further research and operative products schedule

• Scheme:– Some comments on Neural Nets– Histogram matching– Validation / Intercomparison case study:

• Andalusia, Spain: 3 months of 30 minutes rain gauge data for validation

– Global research products• Global IR – derived estimates• METEOSAT - derived estimates

– Further work in this line

Outline Highlights Neural Nets Case Study Products Future work

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

• Highlights

– Why fuse PMW and IR?• Direct response vs indirect relationship• “Bad” spatial and temporal resolutions vs geostationary capabilities• Re-inforce the strengths and avoid the weaknesses

– Inputs processing• IR data from the Global IR database (Janowiak et al 2001) and EUMETSAT archive• PMW Rainfall retrieval based upon Kidd&Barrett SSM/I algorithm:

– V19-V85 or H19-H85 combination over ocean and over land– Polarization Corrected Temperatures (PCT) over coast

• Gauge processing: point to area estimates using maximum entropy interpolation• Histogram matching and GPI calculation for inter-comparison

– Neural nets Inputs selection Model selection Inversion procedures

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Neural Networks

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Neural Networks• NN works fairy well in rainfall estimation

– Operative system: PERSIANN (Sooroshian et al 2000)– Bellerby et al. 2000, etc.

• Neural Nets are not black-boxes– It is possible to make an objective NN selection (Murata et al 1994)– There are inversion procedures to investigate inside – They allow both deterministic and probabilistic approach

• Some advantages over other methods – Any function (Dirichlet’s, not pathological function) can be

approximate with an arbitrary degree of accuracy with a NN: Universal Aproximator.

– An easy method to simulate complex physical models in a quick (operative) way.

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Input selection

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Correlations for some simple models

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

• Several NN architectures Hopfield nets

SOM (cloud characterization)+(GOES data)

Multilayer Perceptron (MLP)

Adaptative Resonance Theory Nets (Grossberg 1969, Carpenter et al 1997) ART1 and ART2

ARTMAP

Distributed ARTMAP

Fuzzy ARTMAP (including a voting procedure (ref))

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Model selection: Results

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Model selection into MLP• Calculate (not guess) the number of neurons in the hidden layer• Network information criterion (NIC) (Murata et al. 1994)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

iLE log2A

ii LLE 'loglog B

n

trwL

nNIC

1

ˆlog1

BA

n Number of observations Set of parameters Gradient2 Hessianlog L Estimated maximum log likelihood

w

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future work

• This allow a conscious design of the net based on Information Theory results

Highlights

• Research after training: model inversion

H

ii

I

j

ihj

hi xwgwf

1 0

, Wxy

*,min oWxfxe

ik

ik

ik x

exx

1

jikx

e

*jjjj ooo

mjOHj

jjj wo ,,

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future work

• What kind of inputs generate an output?: insight into precipitation processes at IR-focus

Highlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Histogram Matching

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Validation

(case study)

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Case study data:– Global IR (Meteosat 5)

– DMSP SSM/I – 30 min gauge validation data

• Resolutions:– Spatial: 4

Km– Temporal: 30 min

• Coverage:– Andalusia (Spain)– Oct-Dec 2001

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future work

• Methodology

Highlights

• What means “field truth” in satellite estimates validation?– Point estimates: more close to the truth AGL– Areal interpolations: encompassing errors and odd effects

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

n

i j

jj

i

iin

jij

nn

zzazzzf

0 02

11

10 2

1exp

2

1,,,

m

innn

m

in

dzdzzzfzzfS1

1111

,,log,,Z

1,,1

111

m

inn

m

in

dzdzzzf

rkcdzdzzzgzzf k

m

innkn

m

in

,,1,,,,1

1111

Con

stra

ints

2) Solving…

1) Maximize the entropy function (using variational methods)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Maximum Entropy InterpolationThe (theoretically) less-biased interpolation method available: an appropriate base to compare

2) Which means that we can solve the computational problems using a simple spherical kriging

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Point measures (average) Maximum Entropy Interpolation

Inverse Distance Weighted

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Small intercomparison of interpolation methods (Niger 2000 and Andalusia 2001)

•IDW

•Bilinear

•Kriging

•MEM

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Inst

anta

neou

s In

terc

ompa

riso

n

SSM/I NN

HM NN*

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Sm

all a

rea,

sho

rt-d

urat

ion

even

tsNal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Instantaneousestimates

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Results: Skill Scores

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Coincident data histogram comparison (October 2001)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• 0.1º Accumulated results

R2 = 0.57

0

10

20

30

40

50

0 10 20 30 40 50

Gauge accumulated at 0.1º (mm/month)

NN

est

ima

te a

t 0.1

º (m

m/m

on

th)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• 0.5º / 3 month accumulated data

R2 = 0.67

0

100

200

300

400

500

600

700

800

0 100 200 300 400 500 600 700 800

Gauge estimate at 0.5º (mm / three months)

NN

est

imat

e at

0.5

º (m

m /

thr

ee m

onth

s)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• 0.5º accumulated results

R2 = 0.73

0

20

40

60

80

100

0 20 40 60 80 100

Gauge accumulated at 0.5º (mm/month) restricted to cells with more than 4 gauge stations in

NN

est

imat

es a

t 0.

5º (

mm

/mon

th)

rest

ricte

d to

cel

ls w

ith

mor

e th

an 4

gau

ge s

tatio

ns in

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Grid size, averaging periods and correlations (Turk et. al 2002)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

Global Coverage

(Reseach Products)

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Global-IR coverage (HM)

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Meteosat coverage (NN)

• Product to be validated using land-GPCC or other dataset

• Oriented to MSG: we are ready to apply this methodology

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

GOES-E 14:32 GOES-E 15:45Trajectories

SSM/I F14 14:30 SSM/I F15 15:44IR temperature along trajectory

•Wind (CMW?) trajectories found by 19x19 correlation matching over 19x19 region. •SSM/I rain then advected along trajectories and adjusted by dIR and tied at end points

•IR/PMW Advection Scheme

• Subscenes:

- Guinea Gulf- GIS integration

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Future operational applications

• QPE / QPF: • SSM/I estimates improve the forecasting (Hou et al 2002) • We can simulate SSM/I

• Agriculture• Hydrology• Natural Hazards

But only when the product become operative and better results will be obtained

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Future research work: MSG and GPM

• Radar data for validation/calibration• Operativity of the global coverage products: intercomparison• Integration in forecasting models: RAMS

• Use of MSG channels:• More information means more discrimination capabilities• Bidirectional reflectance model

• GPM and EGPM addressing

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights

• Conclusions

• Accumulated areal estimates at 0.1º and 0.5º at monthly scale are similar to other works, but the down-top approach allow to know about small scale and short term estimates.• There is an almost-operative product to analyse and to improve with further research.• There are many reseach directions in NN data fusion to follow:

• Inversion• New methods (probabilistic nets)• Integration of other models

• Other physical models can be integrated into the NN methodology.• Any meteorological information can be integrated without major modifications• Complex models can be speed up simulating the result using NN

Nal. Council of Research,Italy

University of L’Aquila,Italy

University of Birmingham,UK

1st INTERNATIONAL PRECIPITATION WORKING GROUP WORKSHOP

Madrid, 23 – 27 September 2002

Outline Neural Nets Case Study Products Future workHighlights