Update to COPC: Global Model Performance Dropouts Dr. Jordan Alpert NOAA Environmental Modeling...
-
Upload
garry-clarke -
Category
Documents
-
view
215 -
download
0
Transcript of Update to COPC: Global Model Performance Dropouts Dr. Jordan Alpert NOAA Environmental Modeling...
Update to COPC:
Global Model Performance Dropouts
Dr. Jordan AlpertNOAA Environmental Modeling Center
contributions from Dr. Brad Ballish, Dr. Da Na Carlis, Dr. Rolf Langland and CDR Mark Moran
2
Outline
• Tools to compare ECMWF and NCEP Dropouts
• Model Runs for Dropout Experiments
• Results from SH Dropout Experiments
• Model Runs for Dropout Experiments
• Comparison of How Well the GSI and ECM “Fit” Obs
• Wind Speed Biases
• How GSI and ECM Fit Observations
3
Poor Forecasts or Skill Score “Dropouts” Lower GFS Performance.
Detailed on next slide
4
Skill Score “Dropouts” LowerOverall GFS Performance.
5
Analysis Team for Dropout Issue
Jordan Alpert - EMCDaNa Carlis - EMCBrad Ballish - NCOKrishna Kumar - NCORolf Langland - NRL
6
Tools to compare ECMWF and NCEP Dropouts
• Use ECMWF analysis to generate “Pseudo Obs” for input to the Gridded Statistical Interpolation (GSI) and GFS forecasts.
• Generate a Climatology of NH and SH dropouts – what are the systematic differences
• Interpolate ECM and GSI analyses to observations to determine comparative strengths and weaknesses.– Statistically analyze observation type fits– stratify by pressure, type, and difference magnitude
Goal: Diagnose Quality Control problems to implement real-time QC detection/correction and improvements to analysis system algorithms
7
Model Runs for Dropout Experiments
• GFS: Operational (Production) GFS
• ECMWF: Operational
• ECM: Pseudo Obs from ECMWF analysis using the GSI as a “Grand Interpolator”
• ecmanlGES: GSI run using ECM as background guess plus GDAS observations
• CNTRL: Updated GSI system with GDAS obs
• InterpGES: Updated GSI with previous 3, 6, & 9-hour ECM forecasts as background guess plus GDAS observations
8
Results from SH Dropout Experiments
• Using ECM (Pseudo Obs) yielded a 90% success rate for alleviating SH dropouts.
• Adding GDAS Obs (ecmanlGES) lowers the success rate to ~75%.• Combining updated GSI system / GDAS obs (CNTRL) with all obs
improves GSI but not enough to alleviate dropouts• Combining background guess from ECM plus all obs (InterpGES )
also improves GSI but not enough to alleviate dropouts• Composites show low-level temperature difference near 850mb in
SH.
Adding the SH observations and the 3, 6, 9-h background guess degrades the ECM forecast
9
Rolf Langland (NRL Monterey)shows systemic height differences between all models and ECMWF (shown is ECMWF-NCEP).
Cause may be the difference in satellite window coverage (under study):
ECMWF (12-h) vs. others (6-h).
Plots at left show height difference plots of time (October to December 2007) vs. longitude, averaged over 35-65N latitudes.
The range of the bias is ±12 m
10
Comparison of How Well the GSI and ECM “Fit” Observations
• Statistics are made on how well the GSI and ECM analyses “fit” the observations for different observation types, as function of pressure, for different regions
• This study shows that each analysis “fits” certain observation types with differing amounts of success, but does not conclude which performs better.
11
Comments on Wind Speed Biases
• The ECM analysis winds at satellite wind locations are stronger than those of the GSI
• The ECMWF data quality control is more aggressive at deleting satellite winds with speeds slower than the background
• ECM wind speeds are stronger as weaker satellite winds are deleted or given less weight in the ECMWF analyses
• For ACARS and sondes, the biases are similar
Work continues to analyze satellite wind speed biases to determine an implementable algorithm for QC, bias correction, and analysis weights
12
How GSI and ECM Fit Observations
• The GSI is found to draw more closely for most winds, especially satellite winds and when there are moderate differences in the analyses (outliers)
• The ECM analyses draw more closely for radiosonde temperatures, especially away from the middle atmosphere and for moderate differences in the analyses
• Fits for surface pressure and moisture are similar – not shown here
• ECM winds are stronger at satellite wind locations based on speed bias stats but with similar fits for radiosonde and ACARS speeds
Changes in the QC and Ob errors could retune these data fits
13
Summary and Future Work
• ECM analysis show dropouts can be alleviated in GFS forecasts
• Running the operational GSI with an ECM derived background guess results in better forecast skill than the operational GFS but not as good as ECM runs
• Running the operational GSI after removing select observation types offers a systematic approach for assessing the impact of different observation types
• There are systematic height differences between all models and ECMWF perhaps from the larger satellite window coverage of ECMWF (12-h) vs others (6-h)
• Work continues to analyze what is the optimal fit of the analysis to observation types and to determine an implementable algorithm for improved QC, bias correction, and analysis weights
• Work has started to use baroclinic instability rates to help analyze dropout cases and analysis differences
14
Background Slides
15
INPUT-----------OUTPUT( )
ECM(WF) Analysis1x1 deg, 15 levels-------------------------
PSEUDO ECM “OBS”
PSEUDO ECM “OBS”GFS GUESS--------------
ECM ANLYSIS“PRE-COND” GUESS
Run GSI
Run GSIagain
ECMWF INITIAL CONDITIONS FOR GFS FORECASTS
“ECM” Runs”
PRE-CONDECM GUESS
PSEUDO ECM “OBS”
---------------ECM ANL
ECM ANL ------------
GFS FORECAST
16
INPUT-----------OUTPUT( )
ECM(WF) Analysis------------------------- PSEUDO ECM “OBS”
PSEUDO ECM “OBS”GFS GUESS--------------
ECM ANLYSIS“PRE-COND” GUESS
Run GSI
Run GSIagain
“InterpGES Runs”
PRE-CONDECM GUESS
PSEUDO ECM “OBS”
---------------ECM ANL
ECM ANL ------------
GFS FORECAST3, 6, 9-hour
GFS FORECASTBKGND GES -3, 0, +3
GDAS Obs ------------
InterpGES Analysis
InterpGES Analysis
------------GFS
FORECAST5-days
17
Initialization Date GFS ECMWF ECM midlatOVRLY polarOVRLY ecmanlGES CNTRL interpGES2008011100 0.68 0.83 0.80 0.76 0.74 0.822008011212 0.69 0.89 0.86 0.83 0.82 0.772008020100 0.68 0.74 0.86 0.852008020300 0.65 0.83 0.83 0.82 0.69 0.632008021700 0.73 0.87 0.89 0.852008030300 0.67 0.82 0.62 0.612008030312 0.59 0.85 0.85 0.84 0.74 0.77 0.69 0.782008030912 0.71 0.91 0.81 0.78 0.69 0.79 0.802008031012 0.71 0.89 0.85 0.75 0.79 0.79 0.772008031300 0.69 0.71 0.83 0.792008031800 0.59 0.79 0.75 0.56 0.69 0.702008031812 0.66 0.63 0.76 0.69 0.78 0.66 0.78 0.752008032012 0.64 0.78 0.75 0.78 0.72 0.702008040900 0.69 0.85 0.84 0.762008042500 0.67 0.77 0.58 0.612008042512 0.67 0.80 0.72 0.72 0.63 0.692008042600 0.61 0.91 0.89 0.68 0.65 0.722008050900 0.68 0.90 0.82 0.802008051000 0.69 0.86 0.83 0.792008051012 0.84 0.90 0.90 0.892008051512 0.65 0.90 0.87 0.782008052200 0.60 0.87 0.84 0.83 0.73 0.702008052212 0.55 0.82 0.75 0.762008061212 0.64 0.85 0.88 0.832008062500 0.70 0.89 0.79 0.752008062512 0.66 0.87 0.77 0.70 0.72 0.772008072500 0.69 0.82 0.812008080500 0.69 0.87 0.892008081600 0.67 0.85 0.862008081912 0.68 0.84 0.82
Excerpt of SH Dropout Climatology List (AC skill scores)
NB: 90% ECM Success in alleviating SH dropout. Adding GDAS Obs (ecmanlGES) lowers the success rate to ~75%. GSI upgrades (CNTRL) improves GSI but not enough to alleviate dropouts, and for InterpGES runs also improves GSI but not enough to alleviate dropouts meaning that adding the SH observations (and the 3, 6, 9-h ges) degrade the ECM forecasts.
18
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
Initialization Date
Southern Hemisphere 5-day Anomaly Correlation Scores (Dropout Cases)
GFS
ECMWF
ECM
CNTRL
InterpECM
19
20
21
22
23
24
25
SH Dropout Composite Cntrl vs InterpGES
Zonal average (above) and mean composite difference between GFS Cntrl and InterpGES (Increments). GFS has low level SH warm “bias” if we consider ECMWF as ground truth.
SH CNTRL 5-day Composite AC = 0.71 SH InterpGES 5-day Composite AC = 0.73
26
Comparison of CNTRL vs. InterpGES Expt.
Dropout Date GFS ECMWF ECM CNTRL INTERPGES
2008042600 0.61 0.91 0.89 0.65 0.72
Guess Analysis
• InterpGES run introduces a lower height and temperature bias to GSI
• ANL differences are not significantly different after GSI interpolation but CNTRL maintains high bias
27
Impact of GSI and OBSThe addition of GSI Observations causes a positive height bias in the SH
CNTRL InterpGES
Dropout Date GFS ECMWF ECM CNTRL INTERPGES
2008042600 0.61 0.91 0.89 0.65 0.72
TypicalDropout case
28
Percent Better Draw for GDAS and ECM Analyses 300-200 hPa 12Z April 2008
0
10
20
30
40
50
60
70
80
90
EU
SA
TW
VA
DW
CA
ND
AR
SO
ND
ES
PR
OF
S
PIB
ALL
S
AIR
EP
AM
DA
R
AC
AR
S
NE
SD
ISW
VW
JMA
IRW
NE
SD
ISIR
W
MO
DIS
WV
WC
T
MO
DIS
WV
WD
L
MO
DIS
IRW
Data Type
Bet
ter
Dra
w in
%
GSI
ECM
EU Satwinds are not used in the GSI
• The GSI draws more formost Satwinds
29
Percent Better Draw for GDAS and ECM Analyses for all Radiosonde Winds 12Z April 2008
0
10
20
30
40
50
60
70
80
SFC-900
900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000
Pressure Category in hPa
Bet
ter
Dra
w in
%
GSI
ECM
The GSI draws more for sonde winds especially away from jet level
30
Percent Better Draw for GDAS and ECM Analyses for Radiosonde Winds with 5-10 m/sec Analysis Difs 12Z April 2008
0
10
20
30
40
50
60
70
80
90
100
SFC-900
900-800
800-700
700-600
600-500
500-400
400-300
300-200
200-100
100-000
Pressure Category in hPa
Bet
ter
Dra
w in
%
GSI
ECM
The GSI draws more for outlier winds – less so around jet level
31
Percent Better Draw for GDAS and ECM Analyses for all Radiosonde Temperatures 12Z April 2008
0
10
20
30
40
50
60
70
SFC-900
900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000
Pressure Category in hPa
Bet
ter
Dra
w in
%
GSI
ECM
The draw for all temperatures is similar – But see next slide
ECM fits better 300 hPa and up
32
Percent Better Draw for GDAS and ECM Analyses for Radiosonde Temperatures with 2-4 Degree Analysis Difs 12Z April 2008
0
10
20
30
40
50
60
70
80
90
SFC-900
900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000
Pressure Category in hPa
Bet
ter
Dra
w in
%
GSI
ECM
The GSI draws more for outlier temperatures in the middle atmosphere and the ECM draws more in the jet level to upper atmosphere
33
Satellite Wind Speed Biases OB-ANL Apr 2008 12Z ECM (Red), GSI (Blue) m/sec
JMA Type 245
-1.2 -0.7 -0.2 0.3
SFC-900
900-800
800-700
700-600
600-500
500-400
400-300
300-200
200-100
JMA IR Type 252
-1.2 -0.7 -0.2 0.3 0.8
700-600
600-500
500-400
400-300
300-200
200-100
MODIS IR Type 257
-1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4
SFC-900
900-800
800-700
700-600
600-500
500-400
400-300
300-200
MODIS WV Type 259
-1.2 -0.7 -0.2 0.3
600-500
500-400
400-300
300-200
Note ECM biases are mostly negative compared to GSI, meaning ECM winds are stronger than GSI winds at satellite wind locations
34
Baroclinicity parameter for GFS and ECMWF
A suitable measure of the baroclinicity is given by the Eady growth rate maximum defined as:
σBI = 0.31 f |(-gp/RT) ∂V/ ∂p| N-1
where f is the Coriolis parameter, V is the total vector wind, N is the Brunt Väisällä frequency and all other parameters have their usual meaning.
This is done to show potential action areas or volatility to propagate initial condition errors into forecast differences
35
NH Eady Stability indexDifference: GFS - ECMWF
SH Eady Stability indexGFS - ECMWF
The predominance of Red over Blue shows GFS has more pronounced baroclinicity compared to ECMWF Ops in low levels (800mb)…
Largest differences for this NHDropout case, 20081021, are in Pacific (disregard most mountain areas as quantities are extrapolated values below Ground).
36
NH GFS Eady Stability index
NH ECMWF Eady Stability index
The total Eady-index shows the Greatest baroclinic potential in the eastern part of the broad Pacific trough, the differences in this case, cause a dropout in the 5-day
forecast.
37
NH Eady Stability indexGFS - ECMWF
SH Eady Stability index GFS - ECMWF
And at 500 mb and 200 mb (not shown), GFS has more pronounced baroclinicity compared to ECMWF Ops. The differences are as much as 20% of the total indexshown on next slide.
Large differences are along trough line with dipole, indicated by Blue and Red indicating difference in position (phase), and large potential for this Rossby wave details.