TIGGE Data Archive and Access TIGGE Data Archive and Access System at NCARSystem at NCAR

5th GIFS-TIGGE Working Group5th GIFS-TIGGE Working Group

South African Weather ServiceSouth African Weather Service

PretoriaPretoria

11-13 March 200811-13 March 2008

Steven WorleySteven Worley

Doug SchusterDoug Schuster

Dave StepaniakDave Stepaniak

-- Others ---- Others --

Topic OutlineTopic Outline

Archive SummaryArchive Summary Provider SummaryProvider Summary Improved Data Handling at NCARImproved Data Handling at NCAR NCAR’s portal and accessNCAR’s portal and access Next Steps - FutureNext Steps - Future

Archive SummaryArchive Summary

Online DataOnline Data– Period, most recent three weeksPeriod, most recent three weeks– ~ 3 TB , public products~ 3 TB , public products– ~ 6 TB, data preparation, subsetting~ 6 TB, data preparation, subsetting

Offline DataOffline Data– Full period of recordFull period of record– ~ 80 TB, NCAR Mass Storage System~ 80 TB, NCAR Mass Storage System

Provide access to bothProvide access to both

Provider SummaryProvider Summary

Geographic and methodsGeographic and methods MetricsMetrics

TIGGE Archive Centers and Data Providers

Archive Centre

Current Data Provider

NCAR NCEP

CMC

UKMO

ECMWFMeteoFrance

JMAKMA

CMA

BoMCPTEC

IDD/LDM

HTTP

FTP

IDD/LDM

Internet Data Distribution / Local Data Manager

Commodity internet application to send and receive data

Summary of Data Providers, March ‘08Summary of Data Providers, March ‘08

Center Conforming Parameters

Ens. Members

Model Res.

Fcst Length

Fcsts/ Day

GB/ Day

Fields/ Day

Files/ Day

ECMWF (ecmf)

70/73 51 N200 (Reduced Gaussian)

10 day 2 115 289,734 328

ECMWF (ecmf)

70/73 51 N128 (Reduced Gaussian)

10-15 day

2 24 138,978 160

UKMO (egrr)

70/73 24 1.25 x 0.83 Deg 15 day 2 21 175,680 488

JMA (rjtd)

61/73 51 1.25 x 1.25 Deg 9 day 1 7 113,192 74

NCEP (kwbc)

59/73 21 1.00 x 1.00 Deg 16 day 4 10 316,596 1040

CMA (babj)

60/73 15 0.56 x 0.56 Deg 10 day 2 28 72,510 82

CMC (cwao)

56/73 21 1.00 x 1.00 Deg 16 day 2 8 163,674 260

BOM (ammc)

55/73 33 1.50 x 1.50 Deg 10 day 2 8 147,972 164

MF (lfpw)

62/73 11 1.50 x 1.50 Deg

2.5 day

1 .15 7,558 33

KMA (rksl)

46/73 17 1.00 x 1.00 Deg 10 day 2 5 64,124 164

CPTEC (sbsj)

55/73 15 1.00 x 1.00 Deg 15 day 2 14 97,084 244

Total 22 240 1,587,102 3,037

Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC

Geopotential Z PL

Specific H PL

T PL

U-velocity PL

V-velocity PL

Potential Vor PT

Potential T PV

U-velocity PV

V-Velocity PV

U 10m SL

V 10m SL

CAPE SL

Conv. Inhib. SL

Land-sea SL

Mean SLP SL

Orog. SL

Skin T SL

Snow D. H20 SL

Snow F. H20 SL

TIGGE Archive Completeness, 02/2008

PL = Pressure Level, PT = 320K Theta Level, SL = Single/Surface Level

Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC

Soil Moist. SL

Soil T SL

Sunshine D. SL

Surf. DPT SL

Surf. ATmax SL

Surf. ATmin SL

Surf. AT SL

Surf. P SL

LW Rad. Out SL

LH flux SL

Net Rad SL

Net Therm. Rad SL

Sensible Rad. SL

Cloud Cov SL

Column Water SL

Precipitation SL

Wilt. Point SL

Field Cap. SL

TIGGE Archive Completeness, 02/2008

PL = Pressure Level, PT = 320K Theta Level, SL = Single/Surface Level

New Data Handling Hardware @ NCARNew Data Handling Hardware @ NCAR

Why ?Why ?– NCEP data processingNCEP data processing

Operational (non-TIGGE) grids via network (59 parameters)Operational (non-TIGGE) grids via network (59 parameters) Create TIGGE compliant grids, mainly metadata, some re-Create TIGGE compliant grids, mainly metadata, some re-

computation (e.g. precip.)computation (e.g. precip.) FTP upload 11 parameters from NCDC - FTP upload 11 parameters from NCDC - work in progresswork in progress Re-distribute to ECMWF and CMARe-distribute to ECMWF and CMA

– Data ingest from all providers, 240 GB/dayData ingest from all providers, 240 GB/day NCAR has maximum task, it produces nothing and receives NCAR has maximum task, it produces nothing and receives allall

– New security model at NCARNew security model at NCAR Forced by TeraGridForced by TeraGrid

– networked based sharing of data and computational cyclesnetworked based sharing of data and computational cycles CostCost

– $120K$120K

The Data Management ChallengesThe Data Management Challenges

Anomalies between centersAnomalies between centers Native horizontal resolutionNative horizontal resolution Number of ensemble membersNumber of ensemble members Number of forecast initialization times (1, 2, 4x daily)Number of forecast initialization times (1, 2, 4x daily) Forecast increment time periodForecast increment time period Forecast lengthForecast length Number of fields providedNumber of fields provided Internal file compression (e.g. jpg) was not specifiedInternal file compression (e.g. jpg) was not specified

Influence on ResearchInfluence on Research

User interface must inform users - commonalities constrain User interface must inform users - commonalities constrain researchresearch

NCAR TIGGE Portal NCAR TIGGE Portal

Address: Address: http:http://tigge//tigge..ucarucar..eduedu

Main FeaturesMain Features– Registration and LoginRegistration and Login– Get DataGet Data– User ToolsUser Tools– Documentation and LinkagesDocumentation and Linkages– Technical and Community Supported HelpTechnical and Community Supported Help

User Interface User Interface

Registration and LoginRegistration and Login– Required per international agreement Required per international agreement

Users electronically accept conditions for usageUsers electronically accept conditions for usage

– We capture metrics for We capture metrics for Name, email, organization name, organization type (univ., Name, email, organization name, organization type (univ.,

gov.,), and countrygov.,), and country Who , what, when files were downloadedWho , what, when files were downloaded

User Interface User Interface

Get Forecast DataGet Forecast Data Two Selection InterfacesTwo Selection Interfaces

– File GranularityFile Granularity Developed FirstDeveloped First

– Parameter GranularityParameter Granularity Added laterAdded later

Dates

Center

File Type

Forecast Time

Forecast Duration

Get Forecast Data

NCAR online file archive

• Selection options•Center(s)•Date•File type (sl, pl, etc)•Initialization time•Forecast length

Download Options• Point and click using browser, one file at a time• Script to run on local machine

•User and password encrypted ‘wget’ commands• background process to access all files

User customized files

• Selection options•Same as for files, plus•Parameter•Regridding•Spatial subsets•Formats, GRIB2, netCDF

Delayed ModeReal Time

Two User Interfaces

ToolsTools

Collection of tools with growing maturityCollection of tools with growing maturity– ContributorsContributors

NCAR NCAR ECMWFECMWF NOAANOAA UnidataUnidata

New API, GRIB2 to netCDFNew API, GRIB2 to netCDF– Developed as ECMWF/NCAR collaborationDeveloped as ECMWF/NCAR collaboration– Need to deploy in the NCAR systemNeed to deploy in the NCAR system

Tools, example; NCAR NCLTools, example; NCAR NCL

QuickTime™ and a decompressor

are needed to see this picture.

User HelpUser HelpTwo modesTwo modes Technical assistance directly from TIGGE staff at NCAR via Technical assistance directly from TIGGE staff at NCAR via

email email – Could originate from the portalCould originate from the portal

Open community website forum, including subscription emailOpen community website forum, including subscription email– Enrollees can post questions, give answers, and share ideas and Enrollees can post questions, give answers, and share ideas and

experiencesexperiences– Provided by UnidataProvided by Unidata

NCAR TIGGE archive usageNCAR TIGGE archive usage

1.1 TB, 95 K files1.1 TB, 95 K files 70 Unique data users70 Unique data users

User Profile

• A few users taking most volume

• 30 UU used parameter subsetting

• Most users take small amounts of data on single visit

Not sufficient for relevant research service

TIGGE data usage TIGGE data usage

Want public access to growWant public access to grow– Web advertisingWeb advertising– Helpful Idea - User Working Group meeting Helpful Idea - User Working Group meeting

in conjunction 3rd THORPEX Science in conjunction 3rd THORPEX Science SymposiumSymposium

THORPEX Exec. Com., 7th February Meeting.THORPEX Exec. Com., 7th February Meeting.

– Publication, possibly EOS/AGUPublication, possibly EOS/AGU– More presentationsMore presentations

Future @ NCAR, Phase I CompletionFuture @ NCAR, Phase I Completion

Finish new system installationFinish new system installation Verify and add NCEP parameters prepared at Verify and add NCEP parameters prepared at

NCDCNCDC Complete advanced subsetting featuresComplete advanced subsetting features

– Spatial, grid interpolation, and user selected output Spatial, grid interpolation, and user selected output format (GRIB2 and NetCDF)format (GRIB2 and NetCDF)

Continue data analysis tool developmentContinue data analysis tool development Quasi-automatic user access to long-term Quasi-automatic user access to long-term

TIGGE holdings from the NCAR MSSTIGGE holdings from the NCAR MSS

Future @ NCAR, add LAMSFuture @ NCAR, add LAMS

Focus on Americas and possibly one or two Focus on Americas and possibly one or two othersothers

Adapt access system for non-global domainsAdapt access system for non-global domains Much to be discussedMuch to be discussed

Future @ NCAR, Phase IIFuture @ NCAR, Phase II

Participation has limiting dependenciesParticipation has limiting dependencies– Finish Phase I Finish Phase I – Demonstrated need from research and education Demonstrated need from research and education

communitiescommunities– Requirements implied by Requirements implied by “data..accessible via

common web user interface”? Single user portal => uniform web service protocol Single user portal => uniform web service protocol

system-wide, interoperable across data, transport, system-wide, interoperable across data, transport, and searchand search

Track development, not contribute resources to R&DTrack development, not contribute resources to R&D

– Need funding path for all but very simple Need funding path for all but very simple adaptationsadaptations

WIS demonstrator project, WMO surplus funds? WIS demonstrator project, WMO surplus funds? THORPEX Exec. Com., 7th February Meeting.THORPEX Exec. Com., 7th February Meeting.

AcknowledgementsAcknowledgements

International data providers, excellent collaboratorsInternational data providers, excellent collaborators– TIGGE compliant data easily entered into the TIGGE TIGGE compliant data easily entered into the TIGGE

archive systemarchive system– Thanks for all the hard workThanks for all the hard work

International Archive Centers, excellent partnersInternational Archive Centers, excellent partners– Very prompt actionsVery prompt actions– Share much software/knowledge/experienceShare much software/knowledge/experience– Demonstrated a new level of International data provisionDemonstrated a new level of International data provision– NCAR has benefited NCAR has benefited

Users have been patient Users have been patient – This has been a Super-sized challengeThis has been a Super-sized challenge

NCAR staff are performing outstandinglyNCAR staff are performing outstandingly

Thanks to SAWS for hosting the 5th GIFS-TIGGE WGThanks to SAWS for hosting the 5th GIFS-TIGGE WG

ENDEND

Steven WorleySteven Worley

worley@ucar.eduworley@ucar.edu