Model Evaluation Tools (MET) - dtcenter.org · ... (software engineer) z [Lacey Holland...
-
Upload
trinhkhuong -
Category
Documents
-
view
220 -
download
4
Transcript of Model Evaluation Tools (MET) - dtcenter.org · ... (software engineer) z [Lacey Holland...
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Model Evaluation Tools (MET)
MET Development TeamDevelopmental Testbed Center
16 July 2008
1
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MET Development Team
Dave Ahijevych (scientist)Barbara Brown (scientist/statistician)Randy Bullock (software engineer) Eric Gilleland (scientist/statistician)John Halley Gotway (software engineer)[Lacey Holland (scientist)]
With thanks to the Air Force Weather Agency (AFWA) and NOAA for their support
2
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Outline
MET Overview – BarbPoint-stat and grid-stat – Barb Confidence intervals – Eric MODE – Randy VSDB and MODE analysis tools – Dave Technical details – John Point- and grid-stat practical – John MODE Tool Practical – John Questions…
3
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MET: A community tool
Goal: to provide a set of forecast verification tools that are
Openly available to the community“Created” by the community, through contributed methods and capabilities
Evaluation methodsGraphical methods
Community includes diverse usersWRF DevelopersDevelopment Testbed Center (DTC)University researchersOperational centers
4
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MET status
MET implementation initiated in fall 2006Version 1.0 released in January 2008Version 1.1 released July 11, 2008
ASCII observationsNeighborhood methodsNew confidence interval estimates for non-Gaussian measures
Version 2.0 in early 2009
5
http://www.dtcenter.org/met/users/
MET isA modular set of verification tools that can be freely downloadedFully documentedSupported through an e-mail help address
6
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Main MET components
Data reformatting modulesStatistics modulesAnalysis modules
7
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Input formatsGRIB model output (WRF post); GRIB gridded analysesASCII or PrepBufr point observations
Re-formattingPCP combine: Adds or subtracts accumulated precipitation from several GRIB files into a single NetCDF fileASCII2NC: Reformats ASCII point obs into the intermediate NetCDF format used by Point-StatPB2NC: Reformats PrepBufr obs into NetCDF format used by Point-Stat
9
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Stats toolsMODE: Method for Object-based Diagnostic Evaluation
Object-based spatial verification approach
Grid-Stat: Compares gridded forecasts and observations
Now includes “neighborhood methods”
Point-Stat: Compares gridded forecasts and point observations (e.g., rawinsonde output)
Includes several methods for matching forecasts to point obs
StatisticsMODE:
Comparisons of object attributesGeometric (size, location, orientation, etc.)Intensity
Grid-Stat and Point-Stat: Traditional stats (POD, FAR; RMSE, MAE, Bias, etc.)Confidence intervals
ParametricBootstrap
10
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Analysis tools:Aggregate and stratify MET output files according to user-specified criteriaReport aggregated statistics and distributions of statisticsAll output in ASCII format
11
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Main MET components
Data reformatting modulesMove data into the format(s) expected by MODE (ascii2nc; pb2nc)Combine precipitation values across time periods (e.g., 24-h totals) (pcp_combine)Subtract precipitation values to create values for finer sub-periods (pcp_combine)
Statistics modulesObject-based spatial verification method (MODE)Verification of grids (Grid-stat)Verification at points (Point-stat)
Analysis modulesAggregate results across cases; stratify results by categoriesVSDB analysis tool (for Grid-stat and Point-stat)MODE analysis tool
12
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Grid-stat and Point-stat scienceStats tools
MODE: Method for Object-based Diagnostic EvaluationGrid-Stat: Compares gridded forecasts and observations
Includes “neighborhood methods”
Point-Stat: Compares gridded forecasts and point observations (e.g., rawinsonde output)
Includes several methods for matching forecasts to point obs
StatisticsGrid-Stat and
Point-Stat: Traditional statistics
Contingency table statistics (POD, FAR, etc.)Continuous statistics (RMSE, MAE, Bias, etc.)
Confidence intervals
ParametricBootstrap
13
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Point Stat: Grid-to-Point Verification
Input Grib forecasts and NetCDFobservations from PB2NCSelect multiple…
Variables, levels, thresholds, masking regions, matching methods, confidence interval (CI) method and alpha values for CIs
Output VSDB and ASCIIContingency table counts and statistics with CIsContinuous statistics with CIsPartial sums
14
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Point-stat: Matching approaches
User-specified region of gridpoints around the observation point:
1 - nearest point2 - 2x2 box around point3 – 3x3 box around pointEtc.
Several metrics can be used to create matched forecast value
Min, Max, Median, Unweighted mean, Distance-weighted mean, Least-squares fit
User-defined min number of valid data points
15
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Grid Stat: Grid-to-Grid verification
Output VSDB and ASCIIContingency table counts and statistics with confidence intervalsContinuous statistics with CIsPartial sums (L1L2, etc.)Neighborhood methods
Output NetCDFMatched pairs and difference fields for each variable, level, masking region
Input Grib or NetCDF from PCP CombineSelect multiple…
Variables, levels, thresholds, masking regions, smoothing methods, confidence interval (CI) methods and alpha values for CIs
16
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Statistics for discrete variables
Measures for 2x2 contingency tablesEx: based on applying a
threshold to a continuous variable
Number of observationsFHO statisticsContingency table countsContingency table proportionsAccuracyBias
Probability of Detection of Yes (PODy)Probability of Detection of No (PODn)False Alarm Ratio (FAR)Critical Success Index (CSI)Gilbert Skill Score (GSS = ETS)Hanssen and KuipersDiscriminant (H-K = TSS)Heidke Skill Score (HSS)Odds Ratio (OR)
17
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Statistics for continuous variables
Forecast/observation meanForecast/observation standard deviation Correlation coefficients (Pearson, Spearman, Kendall’s tau)Mean error (ME)Mean Absolute Error (MAE)Mean Squared Error (MSE)Bias-corrected Mean Squared Error (BCMSE); also known as “standard deviation of the error” (ESTDV)
Root-Mean Squared Error (RMSE)Error percentiles (10th, 25th, 50th, 75th, 90th)
Partial sums (1st and 2nd
moments of the forecasts, observations, and errors)
ScalarAnomalyVectorVector anomaly
18
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Neighborhood verification methods
Also called “fuzzy” verificationUpscaling
Put observations and/or forecast on coarser gridCalculate traditional metrics
Provide information about scales where the forecasts have skill
19
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Neighborhood verification methods
Also called “fuzzy” verificationUpscaling
Put observations and/or forecast on coarser gridCalculate traditional metrics
Provide information about scales where the forecasts have skill
20
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Neighborhood verification methods
Also called “fuzzy” verificationUpscaling
Put observations and/or forecast on coarser gridCalculate traditional metrics
Provide information about scales where the forecasts have skillOutput
Standard contingency table statistics for different scales and thresholdsFractions skill score
21
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Neighborhood verification methods
From Mittermaier 2008
observed forecast
Ebert (2008; Met Applications) describes the neighborhood methods in MET
Example: Fractions skill score (Roberts and Lean, MWR, 2008)
22
Motivation/Background Parametric Intervals Bootstrap Future
Confidence intervals and MET
E Gilleland Confidence Intervals 1 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
23
Motivation/Background Parametric Intervals Bootstrap Future
Background: What is a confidence interval?
Level α hypothesis test vs. (1− α) · 100% confidence intervalInterpretation: if experiment is run 100 times (i.e., 100 CIsestimated), then the true parameter should fall within (1− α) · 100of those limits. For example, if α = 0.05, then we expect, onaverage, that the true parameter would fall inside 95 of theintervals.
E Gilleland Confidence Intervals 2 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
24
Motivation/Background Parametric Intervals Bootstrap Future
Background: Types of confidence intervals used in MET
ParametricNormal approximation
θ ± zα/2 · se(θ)
t-distribution (mean and small samples)
µ± tα/2,n−1 · se(θ)
Nonparametric: IID BootstrapPercentileAdjusted percentile (BCa)
E Gilleland Confidence Intervals 3 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
25
Motivation/Background Parametric Intervals Bootstrap Future
Parametric confidence intervals: Normal Approximation
Assumption is that the sample is independent and identicallydistributed (iid) following, at least approximately, a normaldistribution. For some verification statistics (θ), the normalapproximation can then be used (i.e., θ ± zα/2 · se(θ)).
NotationForecast values are x1, . . . , xn and are iid normal with variance σ2
x
Observations are y1, . . . , yn and are iid normal with variance σ2y
Errors (F −O) are z1 = x1 − y1, . . . , zn = xn − yn and are iidnormal with variance σ2
z
H = hit rate, F = false alarm rate, nH = hits and misses, andnF = false alarms and correct negatives.a, b, c, and d are usual contingency table counts.
E Gilleland Confidence Intervals 4 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
26
Motivation/Background Parametric Intervals Bootstrap Future
Parametric confidence intervals: Normal Approximation in MET
Verification statistics using the normal approximation in MET.The first two also have the t-distribution for small samples.
θ se(θ)
Forecast/Observation Mean (θ = 1n
n∑i=1
xi = x) σx/√
n
Mean Error (θ = x− y = z) σz/√
n
Peirce Skill Score (PSS)√
H(1−H)nH
+ F (1−F )nF
(logarithm of) odds ratio (OR)√
1a + 1
b + 1c + 1
d
E Gilleland Confidence Intervals 5 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
27
Motivation/Background Parametric Intervals Bootstrap Future
Other parametric CIs used in MET
All of these still have an assumption of approximate normality for theunderlying sample, but the intervals derived are not in the same formas the normal approximation intervals.
Forecast/Observation VarianceForecast/Observation standard deviation(Linear) CorrelationHit Rate, False Alarm Rate (Wilson’s interval for proportions)
E Gilleland Confidence Intervals 6 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
28
Motivation/Background Parametric Intervals Bootstrap Future
Bootstrap Confidence Intervals
Assume the sample is iid and representative of the populationResample with replacement from the sample B timesEstimate the statistic(s) for each replicate sample to get a sampleof the statistic(s).Calculate confidence intervals from this sample of the statistic(s)
PercentileBCa (adjusted percentile)
E Gilleland Confidence Intervals 7 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
29
Motivation/Background Parametric Intervals Bootstrap Future
Verification statistics CIs relying on the bootstrap
All statistics in MET can obtain CIs through bootstrap resampling,and at least for the mean, bootstrap CIs are more accurate than thenormal approximation.
E Gilleland Confidence Intervals 8 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
30
Motivation/Background Parametric Intervals Bootstrap Future
Summary and Possible Future Additions to MET
Parametric intervals for: Mean, ME, Variance/std. dev.,Correlation, PSS, log OR, hit/false alarm rates.
Strong assumptions about the samples (iid normal)Fast and easy to computeCan be computed knowing only the single realization of the pointestimate and its standard error
Nonparametric bootstrap intervals for all verification statisticsNo particular population distribution is assumedSample must be iidPotentially more accurate intervals than parametricRequires the entire original sample to computeSlower than parametric intervals
Uncertainty obtained in MET is sampling uncertainty only
E Gilleland Confidence Intervals 9 / 9
c©University Corporation for Atmospheric Research. All rights reserved.
31
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Point Stat
MET Analysis tools and examples
MODE
ASCII
Grid Stat
PSASCII
NetCDF
ASCIINetCDF
GET STATS
OUTPUT
VSDBAnalysis
MODEAnalysis
ASCII
ASCII
AGGREGATE & STRATIFY
GraphicsPackage
GraphicsPackage
User-defined display
46
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MODE Analysis - usage
Usage:mode_analysis -lookin path -summary | -bycase
[-column name] [-dump_row filename] [-out filename] [-help] [MODE file list] [-config config_file] | [MODE LINE OPTIONS]
example:mode_analysis -lookin ./mode_output/wrf2caps/24km -summary \
-column area -column centroid_lat -column axis_ang \-dump_row rowdump.txt \-single -simple \-obs_thr \>=3.000 -area_min 1000
MODEAnalysis
ASCIIMODE ASCII
MODE ASCIIMODE ASCII
MODE ASCIIMODE ASCII
MODE ASCII
9 cases x 3 models x 7 thresholds x 10 conv_radii
47
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MODE Analysis - outputoutput of previous command:
Total mode lines read = 9,352Total mode lines kept = 26
Field N Min Max Mean StdDev
P10 P25 P50 P75 P90 Sum------------
--
-------
-------
-------
------
-------
-------
-------
-------
-------
--------area
26 1098.00 7200.00 2191.77 1357.42 1175.50 1247.00
1720.50 2796.25 3421.00 56986.00centroid_lat
26 31.28 46.48 39.08 3.54 34.38 36.27
39.69 41.26 42.58 1015.95axis_ang
26 -89.83 89.63 26.33 58.82 -67.19 -26.91 58.10 68.69 76.39 684.64
with slightly different options: -summary -column angle_diff
-column interest
-pair -simple -obs_thr
\>3.000 \-interest_min
0.9
Total mode lines read = 9,352Total mode lines kept = 9
Field N Min Max Mean StdDev
P10 P25 P50 P75 P90 Sum----------
-
----
-----
-----
------
----
----
-----
-----
-----
------angle_diff
9 0.53 59.08 21.89 17.76 2.87 8.29 18.42 25.62 45.71 197.04interest
9 0.91 1.00 0.96 0.03 0.93 0.94 0.96 0.99 1.00 8.65
with -bycase
option:
Total mode lines read = 9,352Total mode lines kept = 197
Fcst
Valid Time Area Matched Area Unmatched # Fcst
Matched # Fcst
Unmatched # Obs
Matched # Obs
Unmatched----------------------
------------
--------------
--------------
----------------
-------------
---------------Apr 26, 2005 00:00:00 3576 1257 7 10 2 4May 13, 2005 00:00:00 12546 1592 3 5 2 5...
48
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MODE Analysis – by radius,threshold
convolution radius
prec
ip th
resh
old
1-h precipitation forecast, 24-h lead time, valid 1 June 2005
49
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MODE Analysis – quilt plotPercent observed objects matched - Jun 1, 2005
convolution radius [km]0 20 40 60 80 100 120
prec
ip th
resh
old
[0.0
1”]
50
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MODE object displacement analysis
9 cases from 200548 km convolution radius, 3 mm rain threshold
displacement of composite forecast objects from matched observed objects [degrees]
wrf2caps model wrf4ncar model
mean displacement
51
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
VSDB Analysis
VSDBanalysis
ASCIIGrid Stat ASCII genericgraphicspackage
Gilbert skill score for 7 geometric test cases with 4 rain thresholds
Pearson correlation coefficient for 9 cases from 2005,3 different models with 95% bootstrap confidence intervals
Grid Stat ASCIIGrid Stat ASCII
Grid Stat ASCII
52
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Technical InformationMET distributed as a tarball to be downloaded and compiled locally.
METv1.1 to be released in July, 2008.Updated User’s Guide and Online Tutorial.Register and download: www.dtcenter.org/met/users
Language:Written primarily in C++ with calls to a Fortran library
Supported Platforms and Compilers:Linux machines with GNU compilers
g++ and gfortran or g77Linux machines with Portland Group (PGI) compilers
pgCC and pgf77IBM machines with IBM compilers
xlC and xlf
53
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Dependencies
Required to compile:GNU Make UtilityC++ and Fortran compilers (GNU, PGI, or IBM) NetCDF LibraryBUFRLIB LibraryGNU Scientific Library (GSL)F2C Library (f2c or g2c)
Recommended for use:WRF Post-ProcessorCOPYGB (included with WRF-Post) CWORDSH
54
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Building METSteps for building MET:1. Build the required libraries with the same family
of compilers to be used with MET.2. Select the appropriate Makefile.
GNU, PGI, or IBM3. Configure the Makefile.
C++ and Fortran compilersPaths for NetCDF, BUFRLIB, GSL, and F2C libraries
4. Execute the GNU Make utility.5. Run the test script and check for runtime errors.
Runs each of the MET tools at least once.Uses sample data distributed with the tarball.
55
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
METv1.1 Flowchart
GriddedNetCDF
Point-Stat
GriddedGrib Input:
ObservationAnalyses
ModelForecasts
MODE
PrepBufrPoint Obs PB2NC
PCPCombine
VSDB
Grid-Stat
NetCDFPointObs
PSASCII
NetCDF
ASCIIVSDB
NetCDF
INPUT RFMT INTERMED STATS OUTPUT
= optional
VSDBAnalysis
MODEAnalysis
ASCII
ASCII
AGGREGATE
ASCIIPoint Obs ASCII2NC
56
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Configuration FilesMET is a set of command line tools which are controlled using ASCII configuration files passed to the tools on the command line.Configuration files control things such as:
Fields/levels to be verified.Thresholds to be applied.Interpolation methods to be used.Verification methods to be applied.Regions over which to accumulate statistics.
Well commented and documented in User’s Guide.Easy to modify.Use the version of the configuration files distributed with the tarball.
57
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Use of Configuration Files
GriddedNetCDF
Point-Stat
GriddedGrib Input:
ObservationAnalyses
ModelForecasts
MODE
PrepBufrPoint Obs PB2NC
PCPCombine
VSDB
Grid-Stat
NetCDFPointObs
PSASCII
NetCDF
ASCIIVSDB
NetCDF
INPUT RFMT INTERMED STATS OUTPUT
= optional
VSDBAnalysis
MODEAnalysis
ASCII
ASCII
AGGREGATE
ASCIIPoint Obs ASCII2NC
58
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
PB2NC (18)MODE (65)Grid-Stat (21)Point-Stat (20)VSDB-Analysis (23)MODE-Analysis (89)Only need to modify a few!
Sample File 59
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Data Reformatting Tools
GriddedNetCDF
Point-Stat
GriddedGrib Input:
ObservationAnalyses
ModelForecasts
MODE
PrepBufrPoint Obs PB2NC
PCPCombine
VSDB
Grid-Stat
NetCDFPointObs
PSASCII
NetCDF
ASCIIVSDB
NetCDF
INPUT RFMT INTERMED STATS OUTPUT
= optional
VSDBAnalysis
MODEAnalysis
ASCII
ASCII
AGGREGATE
ASCIIPoint Obs ASCII2NC
60
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
PCP-Combine Tool
Functionality:Sum precipitation across multiple files.New for v1.1:
Add precipitation in 2 files (i.e. NMM output).Subtract precipitation in 2 files (i.e. ARW output).
Specify field name on the command line.No configuration file.
Data formats:Reads GRIB.Writes gridded NetCDF as input to stats tools.
61
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
PB2NC Tool
Functionality:Filter and reformat PREPBUFR point observations into intermediate NetCDF format.Configuration file specifies:
Observation types, variables, locations, elevations, quality marks, and times to retain or derive for use in Point-Stat.
Data formats:Reads PREPBUFR using NCEP’s BUFRLIB.Writes point NetCDF as input to Point-Stat.
CWORDSH utility for FORTRAN blocking
62
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
ASCII2NC Tool (new in v1.1)
Functionality:Reformat ASCII point observations into intermediate NetCDF format.For v1.1, one input ASCII format supported (10 columns):
Message_Type, Station_ID, Valid_TimeLat(Deg North), Lon(Deg East), Elevation(msl)Grib_Code, Level, Height(msl), Observation_Value
No configuration file.Data formats:
Reads ASCII.Writes point NetCDF as input to Point-Stat.Support additional ASCII formats based on user input.
63
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Grid-Stat Tool
Functionality:Computes a variety of statistics for comparing a gridded forecast to a gridded observation.One time step at a time.Data must reside on a common grid.
Recommend COPYGB for regridding GRIB data.Configuration file specifies:
Fields/levels, thresholds, vx regions (grids or polylines)Smoothing options, neighborhood sizesVx methods, confidence interval options, output types
Data formats:Reads gridded GRIB and NetCDF output of PCP-Combine.Writes ASCII statistics and NetCDF matched pairs.
64
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Grid-Stat Usage
Command Line:Required: fcst_file, obs_file, config_fileOptional: -outdir, -v
65
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Grid-Stat Line Types
Statistics line types: 11 possibleCategorical - apply threshold
Contingency table counts (FHO, CTC, CTP, CFP, COP)Contingency table statistics (CTS)
Continuous - raw fieldsContinuous statistics (CNT)Partial Sums (SL1L2)
Neighborhood – choose size (new for v1.1)Neighborhood categorical (NBRCTC, NBRCTS)Neighborhood continuous (NBRCNT)
Ten header columns common to all line types.Data in remaining columns specific to each line type.
66
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Grid-Stat Output
Output files:ASCII statistics file containing all line types (File ends with “.vsdb”).Optional ASCII files sorted by line type with a header row (File ends with “_TYPE.txt”).Optional NetCDF matched pairs and difference fields. (File ends with “_pairs.nc”).
Naming conventions:grid_stat_HHMMSSL_YYYYMMDD_HHMMSSV[.vsdb | _pairs.nc | _TYPE.txt]Ex: grid_stat_120000L_20050807_120000V.vsdb
67
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Point-Stat Tool
Functionality:Computes a variety of statistics for comparing a gridded forecast to point observations.One time step at a time.Configuration file specifies:
Fields/levels, thresholds, vx regions (grids, polys, stations)Vx message types, interpolation methodsVx methods, confidence interval options, output types
Data formats:Reads gridded GRIB and NetCDF output of PCP-Combine.Reads point NetCDF output of ASCII2NC and PB2NC.Writes ASCII statistics.
68
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Point-Stat Usage
Command Line:Required: fcst_file, obs_file, config_fileOptional: -climo, -ncfile, -outdir, -v
69
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Point-Stat Line Types
Statistics line types: 12 possibleCategorical - apply threshold
Contingency table counts (FHO, CTC, CTP, CFP, COP)Contingency table statistics (CTS)
Continuous - raw fieldsContinuous statistics (CNT)Partial Sums (SL1L2, SAL1L2, VL1L2, VAL1L2)
Matched Pairs (new for v1.1)Raw matched pairs – a lot of data! (MPR)
Ten header columns common to all line types.Data in remaining columns specific to each line type.
70
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: Point-Stat Output
Output files:ASCII statistics file containing all line types (File ends with “.vsdb”).Optional ASCII files sorted by line type with a header row (File ends with “_TYPE.txt”).
Naming conventions:point_stat_HHMMSSL_YYYYMMDD_HHMMSSV[.vsdb | _TYPE.txt]Ex: point_stat_120000L_20050807_120000V.vsdb
71
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE Tool
Functionality:Identifies objects in two fields, computes single object attributes and pair-wise differences, matches/merges objects, writes object attributes and differences.One time step at a time.Data must reside on a common grid.
Recommend COPYGB for regridding GRIB data.Configuration file specifies:
Forecast and observation specified separatelyField/level, raw filter value, mask region (grid, polyline)Object definition parameters (convolution radius and threshold)Object filtering parameters (area, intensity)
Flags for matching/merging logicFuzzy engine weights, interest functions, and confidence mapsTotal interest thresholdPostScript plotting options
Data formatsReads gridded GRIB and NetCDF output of PCP-Combine.Writes ASCII statistics, NetCDF object fields, PostScript summary plot.
72
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE Usage
Command LineRequired: fcst_file, obs_file, config_fileOptional: -config_merge, -outdir, -vDisable output: -plot, -obj_plot, -obj_stat, ct_stat
73
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE Output
Output files:ASCII object statistics file (File ends with “_obj.txt”).ASCII Contingency table statistics file (File ends with “_cts.txt”).NetCDF object file (File ends with “_obj.nc”).PostScript summary plot (File ends with “.ps”).
Naming conventions:mode_FFIELD_FLVL_vs_OFIELD_OLVL_HHMMSSL_YYYYMMDD_HHMMSSV_HHMMSSA [.obj.txt | _cts.txt | _obj.nc | .ps]Ex: mode_APCP_12_SFC_vs_APCP_12_SFC_120000L_20050807_120000V_120000A_obj.txt
74
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE Object Stats
Four object statistics line types (contents of OBJECT_IDcolumn):
Simple forecast and observation objects (FNNN and ONNN)Pairs of simple objects (FNNN_ONNN)Composite forecast and observation objects (CFNNN and CONNN)Pairs of composite object (CFNNN_CONNN)
Same number of columns for each line type (50 in total):18 header columns.20 columns applicable to SINGLE simple and composite line types.12 columns applicable to PAIRS of simple or composite line types.Columns which do not apply to a given line type contain fill data (-9999)
May be disabled using the –obj_stat command line argument.
75
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE CTStats
Contains traditional contingency table counts and corresponding statistics computed in three ways:
Scoring the RAW fields by applying the convolution thresholds.Scoring the FILTERed fields by first applying any raw filters and then applying the convolution thresholds.Scoring point-wise using the resolved OBJECT fields.
Meant simply as a point of reference for the MODE method.Differs from the VSDB CTS line type.
Does not include confidence intervals.May be disabled using the –ct_stat command line argument.
76
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE NetCDF
NetCDF output file contains 4 fields:Indices for the simple forecast objects.Indices for the simple observation objects.Indices for the composite forecast objects.Indices for the composite observation objects.
May be disabled using the –obj_plotcommand line argument.
77
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
Practical: MODE PostScript
MET does not generally provide plotting tools.Exception for MODE to illustrate the method.Configuration file plotting options:
Specify colortables to be used for plotting raw and object fields.61 colortables provided in data/colortables.Specify how to rescale an existing colortable.Or explicitly define you own.
Option for how colorbar is plotted.Draw lines as great circle arcs or straight lines in the grid.Zoom plot up to only the valid region of data.
Number of pages of PostScript output based on configuration fileselections:
At least 4 depicting object definition and matching.Additional pages for:
Merging using the double-threshold technique.Merging using the fuzzy engine technique.
78
Copyright 2008, University Corporation for Atmospheric Research, all rights reserved
MODE PS Pages 1 and 2 79