Near real time assessment of operational oceanography ...godae-data/Symposium/GOV...Multimodel...

GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013


Near real time assessment of operational oceanography products:

advances in the GOV community, overview of Class 4 intercomparison

and multimodel assessment approaches

Greg Smith Environment Canada

F. Hernandez Mercator-Ocean

M. Martin, A. Sellar UK Metoffice

With thanks to the many IV-TT contributors!

www.godae-oceanview.org


• Pursues activities developed during GODAE

• Develop a framework for comparing outputs of the

various operational ocean forecasting systems (OOFS)

– leads to improvements to the systems and to the quality of products

from those systems

– provides a framework for scientific discussions on forecasting system

performance assessment, on ocean analysis and validation from

numerical simulations

– offers a demonstration of the work in GODAE OceanView and has the

potential to increase the visibility to the external community

The Intercomparison and Validation Task Team:


The Intercomparison and Validation Task Team

Coordinates and promotes the development of scientific validation and intercomparison of operational oceanography systems – Mainly for short term forecast assessment, but benefit for/from

reanalysis assessment, and medium range forecast assessment (eg, seasonal forecast),

– include the definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skills) both for physical and biogeochemical parameters and the setting up of specific global and regional intercomparison experiments

– metrics related to specific applications also considered

– links with the OSE-TT

– links with the JCOMM ET-OOFS team for operational implementation

– cooperation with CLIVAR/GSOP for climate issues.


• Main outcomes from the IV-TT: – Class 4 metrics, forecast skill monitored in near real time

among 6 operational centres

– Class 1 metrics, ensemble and multi-model approach

– Participation to the GSOP Intercomparison project (CLIVAR/GOV initiative, presented by M. Balmaseda)

• Advances in real time monitoring and validation methodology implemented in operational oceanography centres: highlights with some examples

• How operational validation is provided to users in term of Product Quality: some illustration and examples

• Conclusions

Outline of this talk


Class 4 metrics

• Create a real-time system for generating model-equivalents to a common set of observations

• Major effort made to ensure consistency in methods and comparisons

• Initial comparison: – Jan-Jun 2013, 6 institutions participated

• Systems evaluated using: – SST from drifting buoys – Argo temperature and salinity profiles – Sea level anomaly

• Statistics compiled for analyses as well as as a function of lead time


USGODAE – In-situ drifters Courtesy Andy Ryan, UK Metoffice


ARGO – Salinity profiles


Taylor diagram of SST around Australia

• Taylor diagram provides another perspective

See poster from Divakaran for details


Class4 intercomparison

• Radar plots of RMS error computed for the period of 1st January to 30th September 2013, for SLA, SST and in the layer 0-500m for temperature and salinity

• Systems involved : PSY3 (Mercator OCEAN), FOAM (UK Met Office), GIOPS (CMC), RTOFS(NOAA/NCEP), OceanMAPS (Bureau Of Meteorology)

• For salinity PSY3 shows the best performance

• For temperature, except OceanMAPS (lower resolution except in Australian region), four other systems are very close

• FOAM SST is the most accurate system with GIOPS approaching in terms of forecast

See poster from C. Regnier for details


Mapping of RMS error

• We are working with the July-August-September 2013 quarter with the five systems

• The ocean is divided in regular bins of 2°x2°

• In the left we show the system for which the RMS error is the lowest

• In the right the corresponding RMS value is displayed for each bin

• Maps exhibit a regional distribution of the error

• This kind of map is a way to compare system with a given criterion

• Need to cross several criteria for robust conclusion

See poster from C. Regnier for details


Class 1 Multi-model and ensemble validations


Multimodel ensemble SST validation

• Demonstrates value of multimodel ensemble

• Results sensitive to method used to calculate ensemble mean:

– simple

– weighted

– kmeans

3. Time series of RMS and bias for all members and ensembles for period 8 June 2013 to 8 July 2013

See poster from T. Spindler for details


CLASS1: drifter trajectory forecast

• How can we best use OOFS to determine the position of the AF447 crash?

• Use 15m drifter trajectories to estimate error from 6 systems

• Demonstrates added value of multimodel ensemble

7-day fcst 3-day 1-day

1-day forecast: in 50% of the forecasts, the distance between the drifter and the end point trajectory is less than 20km

~20 km


Evaluating oil-spill dispersion

forecasting in the Northern

Aegean Sea

See poster by S. Sofianos et al. for details

Highlights the importance of ensemble methods!


Sea ice forecast verification using analyses is unreliable due to large analysis uncertainty

• Patchy coverage, foot-print issues, data reliability

• Representation of leads

Only evaluate points where the analysis changes by more than 10%

Pro: Only includes points where we have confidence in ice analysis

• Focus on ice edge in particular

Con: Excludes areas of incorrect model changes

• E.g: coastal polynyas, false alarms along the ice edge

Sea ice verification: Analysis tendencies

7day GIOPS forecast

error for 2011-03-30

Van Woert et al. (2004)

See poster from G. Smith for details


Sea ice verification : Analysis tendencies

• Over a full annual cycle, this method provides a reasonably reliable idea of forecast skill

• But what about leads, coastal polynyas and false alarms in the marginal ice zone?

GIOPS 7 day forecasts for 2011

GIOPS Analysis persistence

Van Woert et al. (2004)


Contingency table analysis

• Comparison with IMS Analyses:

– Interactive Multisensor Snow and Ice Mapping System (NOAA-NIC)

– Daily Northern Hemisphere ice analyses on 4km grid (ice/water)

• Evaluation Methodology:

– Calculate contingency table values using 0.4 ice concentration cutoff

IMS Ice IMS No ice

Forecast

Ice

Hit ice False

Alarm

Forecast

No ice

Miss Hit water

• Proportion Correction Ice:

PCI = Hit ice / (Hit ice + Miss) [0,1]

• Proportion Correction Water:

PCW = Hit water / (Hit water + False Alarm) [0,1]


Contingency table verification

• Proportion Correct Ice

– Ability to forecast ice formation and advection

– PCI= Hits ice / (Hits + Misses)

– Shows skillful forecasts along most ice edges

• Proportion Correct Water

– PCW= Hits water / (Hits water + False Alarms)

– Shows important false alarm rate missing from analysis tendency verification

ΔPCI ( Fcst – Pers)

ΔPCW ( Fcst – Pers)

GIOPS 7 day forecasts for 2011

Ice? YIMS NIMS

YFcst Hit ice False

Alarm

NFcst Miss Hit

water

Skill

Error

See poster from G. Smith for details


BAL MFC: Developing ice edge metrics for the

Baltic Sea

GODAE OceanView Symposium – November 2013

obs2mod

distance

mod2obs

distance

Observed ice concentration

obs ice edge

model ice edge min distance for

each edge point

N

i

iDN

RMSdist1

21modnnN obs

mod_mod,21mod,2_,2mod1,2mod ...... nobsobsobsnobsobsi ddddD

Metric must be calculated sub-regionally to avoid long overland distances

Ice edge RMS distance

Find minimum distance from modeled ice edge for each observed ice edge cell and vice versa

Take the RMS of these distances

Straightforward interpretation for the users

Courtesy of P. Lagemaa


Conclusions

• Direct benefit from IV-TT activity:

– several OOFS have now implemented routine Class-4 and Class-1 delivery, allowing enhanced monitoring for these teams, and definition of new validation approaches

– Operational oceanography validation experts organized similar to that in the atmospheric community


Conclusions (con’t)

• General overview:

– Validation techniques are implemented in most OOFS, global or regional

– New metrics are defined to satisfy specific user needs and down stream applications


Thanks!

What is the role of GOV in improving Polar Predictions?


Sea Level Forecast Skill at Mercator Océan

Improvement of the forecast / persistence

Global:

¼° old

¼° new

Zoom:

1/12° old

1/12° new

• Implemented routinely, based on Class-4 metrics

• Statistics published every 3 months • Used here to compare former and new

operatioanl system


Communicate Product Quality to users:

• Regular reporting:

– QuOVaDis quarterly report by Mercator Océan

– MyOcean2 Quality Information Docucments, complemented by quarterly Class-4 reporting

– …..

• Real time reporting

– Website


« NARVAL »: regional validation webserver for the

MyOcean2 IBI area

n 2 NARVAL modes

n On-line “Real-time” mode: Val procedures launched after daily cycle

n Delayed mode Component: Val procedures on specific metrics and statistics covering longer periods (i.e. monthly, seasonal and annual)

n NARVAL updated to generate automatically Statistics for MyO Quarterly Validation Reports.

n Intercompares with same metrics available MyO fcst in same area. eg, 3 in the western Med sea (MFS, Mercator, IBI)

Courtesy of Puertos del Estado and MyOcean2.

MyOcean2

V3 Acceptance & V4 Product Design Workshops – Madrid 28,29th January 2013

Partial snapshots of NARVAL Real-Time & Delayed-Modes web pages


Inter-comparison partners

Institution System Ocean

Model

Resolution

UK Met Office FOAM NEMO ¼ degree

Bureau of Meteorology OceanMAPS mom4p1 1 degree (1/10 around Australia)

NOAA/NWS/NCEP/EMC/MMAB RTOFS HYCOM 1/12 degree

Mercator-Ocean PSY3 NEMO ¼ degree

Canadian Meteorological Centre CONCEPTS-GIOPS NEMO ¼ degree

REMO HYCOM ATLe0.25 ¼ degree (Atlantic only)

•Results available for all partners for January-June 2013


ARGO – Temperature profiles


ARGO – Salinity profiles


CLASS 1: drifter trajectory forecast

Scientific question: How to make the best trajectory forecast in the context of operational oceanography ?

6 products used: • GLORYS • PSY2 • PSY3 • NCOM • HYCOM • CLS


CLASS 1: drifter trajectory forecast

•

•

•



7-day fcst 3-day 1-day

1-day forecast: in 50% of the forecasts, the distance between the drifter and the end point trajectory is less than 20km

~20 km



Main conclusions: • Trajectory forecasts is a nice way to evaluate OOFS skill • The added value of an ensemble of forecasts is shown • The ensemble filters out the error

Smoothing increases

Erro

r R

MS

(km

)

Near real time assessment of operational oceanography ...godae-data/Symposium/GOV...Multimodel...

Documents

Transcript of Near real time assessment of operational oceanography ...godae-data/Symposium/GOV...Multimodel...