Project CRAFT: A Test BedProject CRAFT: A Test Bedfor Demonstrating the Realfor Demonstrating the Real
Time Acquisition and AcrhivalTime Acquisition and Acrhivalof WSR-88D Base (Level II) of WSR-88D Base (Level II)
DataData
Presented to UCAR/Unidata ProgramPresented to UCAR/Unidata Program
12 October 2000 12 October 2000
Kelvin K. DroegemeierKelvin K. DroegemeierCenter for Analysis and Prediction of Storms Center for Analysis and Prediction of Storms
andandSchool of MeteorologySchool of MeteorologyUniversity of OklahomaUniversity of Oklahoma
s fn
NEXRAD Doppler Radar NetworkNEXRAD Doppler Radar Network
NEXRAD Facts and FiguresNEXRAD Facts and Figures
158 radars (141 in the Continental US)158 radars (141 in the Continental US)– 120 National Weather Service radars120 National Weather Service radars– 26 Department of Defense radars26 Department of Defense radars– 12 Federal Aviation Administration radars12 Federal Aviation Administration radars
s fn
WSR-88D ComponentsWSR-88D Components
Radar DataRadar DataAcquisitionAcquisition
s fn
WSR-88D ComponentsWSR-88D Components
Radar ProductRadar ProductGeneratorGenerator
s fn
WSR-88D ComponentsWSR-88D Components
Principal UserPrincipal UserProcessorProcessor
s fn
WSR-88D ComponentsWSR-88D Components
Principal UserPrincipal UserProcessorProcessor
NEXRAD Data TypesNEXRAD Data Types
Archive Level I (analog, time domain output)Archive Level I (analog, time domain output) Level II data (digital base data at full resolution)Level II data (digital base data at full resolution) Archive Level III (“NIDS” digital products from the Radar Product Archive Level III (“NIDS” digital products from the Radar Product
Generator)Generator) Archive Level IV (at the PUP)Archive Level IV (at the PUP)
NEXRAD Data TypesNEXRAD Data Types
Archive Level I (analog, time domain output)Archive Level I (analog, time domain output) Level II data (digital base data at full resolution)Level II data (digital base data at full resolution) Archive Level III (“NIDS” digital products from the Radar Product Archive Level III (“NIDS” digital products from the Radar Product
Generator)Generator) Archive Level IV (at the PUP)Archive Level IV (at the PUP)
s fn
NEXRAD Level III (NIDS) NEXRAD Level III (NIDS) DataData
24 products available from all CONUS 24 products available from all CONUS radars in real timeradars in real time
Provided by 3 commercial vendorsProvided by 3 commercial vendors Lowest 4 elevation angles of the radial Lowest 4 elevation angles of the radial
velocity and reflectivityvelocity and reflectivity– 1 degree x 1 km resolution1 degree x 1 km resolution– Values are quantized (e.g., 0-5, 5-10, 10-Values are quantized (e.g., 0-5, 5-10, 10-
15)15)
s fn
NEXRAD Data TypesNEXRAD Data Types
Archive Level I (analog, time domain output)Archive Level I (analog, time domain output) Level II data (digital base data at full resolution)Level II data (digital base data at full resolution) Archive Level III (“NIDS” digital products from the Radar Product Archive Level III (“NIDS” digital products from the Radar Product
Generator)Generator) Archive Level IV (at the PUP)Archive Level IV (at the PUP)
s fn
NEXRAD Base (Level II) NEXRAD Base (Level II) DataData
Full resolution digital dataFull resolution digital data– Full data precisionFull data precision– All tiltsAll tilts– Not quality controlledNot quality controlled
1 degree x 1 km resolution for 1 degree x 1 km resolution for reflectivityreflectivity
1 degree by 0.25 km resolution for radial 1 degree by 0.25 km resolution for radial windwind
Not available in real time except for Not available in real time except for selected sites (more on that later)selected sites (more on that later)
s fn
Base Data Usage: NSSL Warning Base Data Usage: NSSL Warning Decision Support System on 3 May Decision Support System on 3 May
19991999
Courtesy National Severe Storms Laboratory
Trimmed Detections and Ground Truth Damage Paths
Hits (142)
Misses (25)
FAs (21) Courtesy D. Zittel
5 April 1999 - Impact of Base Data5 April 1999 - Impact of Base Data
s fn
12 Z Reflectivity
Initial 700 mb VerticalVelocity Using NIDS
Initial 700 mb VerticalVelocity Using Level II
Data and SDVR
Courtesy S. Weygandt and J. Levit
5 April 1999 - Impact of Base Data5 April 1999 - Impact of Base Data
s fn
15 Z Reflectivity
3 hr ARPS CREF Forecast (9 km) Using Level II
Data and SDVRValid 15Z
3 hr ARPS CREF Forecast (9 km) Using
NIDS DataValid 15Z
Courtesy S. Weygandt and J. Levit
s fn
Recording Level III WSR-Recording Level III WSR-88D Data88D Data
Limited set of products recorded on Limited set of products recorded on CD-ROM at the radar siteCD-ROM at the radar site
s fn
WSR-88D Level III Data WSR-88D Level III Data Archival at NCDCArchival at NCDC
StationsStations 120120MediaMedia CDCDBytes/mediaBytes/media 600 MB600 MBDaily ReceiptsDaily Receipts 4 GB4 GBAnnual ReceiptsAnnual Receipts 60 TB60 TB
Courtesy Steve DelGreco (NCDC)
NWS
RecordDisks
Disks
To NCDC
PUP
Archive
Disk Storage
PC
Disk DriveCopy Files to HDSS
HDSS
WSR-88D Level III Data WSR-88D Level III Data ProcessingProcessing
s fn
Level III Requests at NCDCLevel III Requests at NCDC
s fn
Recording WSR-88D Level Recording WSR-88D Level II DataII Data
Not in original plansNot in original plans Exabyte 8 mm tape recorders added Exabyte 8 mm tape recorders added
to all radars (1994) as an interim to all radars (1994) as an interim solutionsolution
s fn
WSR-88D Level II Data WSR-88D Level II Data Archival at NCDCArchival at NCDC
StationsStations 158158MediaMedia 8 mm tape*8 mm tape*Bytes/mediaBytes/media 4.8 GB4.8 GBDaily ReceiptsDaily Receipts 165 GB165 GBAnnual ReceiptsAnnual Receipts 60 TB60 TB
*6-step process with NCDC operating 92 Exabyte *6-step process with NCDC operating 92 Exabyte tape drivestape drives
Courtesy Steve DelGreco (NCDC)
WSR-88D Level II Data WSR-88D Level II Data ProcessingProcessing
s fn
Level II Requests at NCDCLevel II Requests at NCDC
s fn
WSR-88D Data Archival at WSR-88D Data Archival at NCDCNCDC
% Rcpts % Rcpts #Sites#Sites
Level II NWSLevel II NWS 64.964.9 120120Level II DoDLevel II DoD 35.735.7 2626Level II FAALevel II FAA 9.59.5 1212
Total 88D NetworkTotal 88D Network 55.755.7 158158
Level III NWSLevel III NWS 89.589.5 120120
s fn
Currently 42,000 8-mm tapes in archiveCurrently 42,000 8-mm tapes in archive 9,500+ copies distributed to date9,500+ copies distributed to date Cost per tape is nominal ($100 for a radar day) Cost per tape is nominal ($100 for a radar day)
but high for large (multi-year) data setsbut high for large (multi-year) data sets Major time delays (2-4 months; human Major time delays (2-4 months; human
resources)resources) Archive incompleteArchive incomplete No mechanism for perusing the data baseNo mechanism for perusing the data base NOAA spent $768,000 on Level 2 recording NOAA spent $768,000 on Level 2 recording
system repairs from 6/1/98-6/14/99system repairs from 6/1/98-6/14/99 NCDC maintains 92 Exabyte tape systemsNCDC maintains 92 Exabyte tape systems
WSR-88D Base Data Archival WSR-88D Base Data Archival at NCDCat NCDC
s fn
Summary: WSR-88D Radar DataSummary: WSR-88D Radar Data The scientific and operational communities need The scientific and operational communities need
base data (real time and archived)base data (real time and archived) Although NIDS data are available in real time from Although NIDS data are available in real time from
allall WSR-88D radars, they are insufficient for many WSR-88D radars, they are insufficient for many applications (NWP, hydrology)applications (NWP, hydrology)– Degradation of precisionDegradation of precision– Only the lowest 4 tilts are transmittedOnly the lowest 4 tilts are transmitted
Base data currently are Base data currently are notnot available in real time available in real time– Originally would have been expensive Originally would have been expensive – Presumed large volume of data (10 mbytes/5 min/radar)Presumed large volume of data (10 mbytes/5 min/radar)– Need wasn’t there 10 years agoNeed wasn’t there 10 years ago
The technology and need now exist to prototype the The technology and need now exist to prototype the direct acquisition, use, and archival of base data in direct acquisition, use, and archival of base data in real timereal time
s fn
The Collaborative Radar The Collaborative Radar Acquisition Field Test Acquisition Field Test
(CRAFT)(CRAFT) Establish a prototype real time WSR-88D base Establish a prototype real time WSR-88D base
data acquisition test bed todata acquisition test bed to– Evaluate strategies for Evaluate strategies for compressing and transmittingcompressing and transmitting
base data in real timebase data in real time– Develop efficient and cost-effective strategies for Develop efficient and cost-effective strategies for
direct digital ingest, archive, and retrievaldirect digital ingest, archive, and retrieval at NCDC at NCDC– Assess the value of base data in Assess the value of base data in numerical weather numerical weather
predictionprediction– Test web-based Test web-based data miningdata mining techniques for rapid techniques for rapid
perusal/access of base data by the scientific perusal/access of base data by the scientific communitycommunity
CRAFT was established by the OU Center for CRAFT was established by the OU Center for Analysis and Prediction of Storms (CAPS) in Analysis and Prediction of Storms (CAPS) in collaboration with the collaboration with the – UCAR Unidata Program OfficeUCAR Unidata Program Office– NOAA WSR-88D Operational Support FacilityNOAA WSR-88D Operational Support Facility– University of WashingtonUniversity of Washington
CAPS and the NOAA National Severe Storms CAPS and the NOAA National Severe Storms Laboratory are now jointly coordinating the Laboratory are now jointly coordinating the effort in conjunction with the NEXRAD OSF effort in conjunction with the NEXRAD OSF and UCARand UCAR
s fn
ParticipantsParticipants
RIDDS: The Foundation of RIDDS: The Foundation of CRAFTCRAFT
s fn
s fn
Technical StrategyTechnical Strategy
Repeater Hub
RIDDS Linux PC
Unidata LDM ($1500)
Server
Dedicated 56K line
($2000 - $6000/year)Cisco 1600
Series Router($2000)
WSR-88D
At the radar site
Users
Internet
s fn
Original CRAFT NetworkOriginal CRAFT Network
Sharing Data from Multiple Sharing Data from Multiple Sources via Cooperating Sources via Cooperating
LDMsLDMs
Source
LDM
Source
Source
LDM LDM
LDMLDM
LDM LDM
LDM
LDM
Internet
Courtesy D. Fulker, Unidatas fn
– We added to LDM in-line We added to LDM in-line compression/decompression (BZIP2 off-the- compression/decompression (BZIP2 off-the- shelf algorithm)shelf algorithm)
maximum 16:1 in clear airmaximum 16:1 in clear air about 8:1 in the worst caseabout 8:1 in the worst case average of about 12:1average of about 12:1
– Maximum aggregate data rate is 15 Maximum aggregate data rate is 15 Mbytes/radar/5 min uncompressed = 48 Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for Mbits/sec compressed for entire national entire national networknetwork
bandwidth isn’t the issue, but rather topology, bandwidth isn’t the issue, but rather topology, quality of service, and data storagequality of service, and data storage
one year’s worth of compressed data from NWS one year’s worth of compressed data from NWS radars is about 10 petabytesradars is about 10 petabytes
s fn
Technical HighlightsTechnical Highlights
– Compression now being improved (hope Compression now being improved (hope for > 20:1)for > 20:1)
– LDM relays data to other nodesLDM relays data to other nodes unlimited system growthunlimited system growth redundancyredundancy
– Simultaneous data transmission to multiple Simultaneous data transmission to multiple usersusers
– Single point of failure (modem is backup)Single point of failure (modem is backup)
s fn
Technical HighlightsTechnical Highlights
s fn
Real Time Management and Reliability: Real Time Management and Reliability: >95%>95%
http://traffic.onenet.net/
Real Time Management and Reliability: Real Time Management and Reliability: >98%>98%
Steve Goodman, NASA/MSFC
s fn
CRAFT Phase I: Proof-of-CRAFT Phase I: Proof-of-ConceptConcept
Internet2 is ...Internet2 is ...– a project, led by universities, to develop a project, led by universities, to develop
advanced internet technology and advanced internet technology and applications vital to the research and applications vital to the research and education missions of higher educationeducation missions of higher education
– big commitmentbig commitment member universities have committed over $70 member universities have committed over $70
million per year in new investment on their million per year in new investment on their own campuses own campuses
corporate members have committed more corporate members have committed more than $30 million over the life of the projectthan $30 million over the life of the project
Internet2 and AbileneInternet2 and Abilene
s fn
Abilene is ...Abilene is ...– an advanced an advanced backbone network testbedbackbone network testbed
that connects regional gigaPoPsthat connects regional gigaPoPs supports Internet2 applications developmentsupports Internet2 applications development that is focused exclusively on university research and that is focused exclusively on university research and
education education that is developing next generation operational and that is developing next generation operational and
quality of service capabilitiesquality of service capabilities
Internet2 and AbileneInternet2 and Abilene
s fn
Abilene NetworkAbilene NetworkJanuary 1999 January 1999
Cleve
land
New York
Atlanta
Indianapolis
Kansas City
Houston
Denver
Los Angeles
Sacramento
Seattle
Abilene Router Node
Abilene Access Node
Operational January 1999
Planned 1999
s fn
New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD
s fn
New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD
s fn
New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD
s fn
NCEP
NCDC
OU
NCAR/FSL
U-WA
New Concept: Abilene/Internet2 + New Concept: Abilene/Internet2 + NEXRADNEXRAD
AWC
TPC
Abilene Backbone
Other LDMs(university, private,
government)
NCARLDM
NCEPLDM
FSLLDM
NCDCLDM
OULDM
U-WALDM
Via Abilene orCommodity Internet
56K PhoneLines to Abilene
Universities
Logical Network TopologyLogical Network Topology
s fn
Adapted from D. Fulker, UCAR
s fn
Alternative StrategyAlternative Strategy
Repeater Hub
RIDDS Linux PC
Unidata LDM ($1500)
Server
Internet Direct
Cisco 1600 Series
Router($2000)
WSR-88D
At the radar site
Users
Internet
Abilene Backbone
Other LDMs(university, private,
government)
NCARLDM
NCEPLDM
FSLLDM
NCDCLDM
OULDM
U-WALDM
Via Abilene orCommodity Internet
Direct Connectionto the Internet inthe NWS Office
Alternative Network TopologyAlternative Network Topology
s fn
Adapted from D. Fulker, UCAR
– Configuration completely scalableConfiguration completely scalable– EveryoneEveryone has real time access to WSR-88D has real time access to WSR-88D
base data (all radars)base data (all radars) Special quality of service for key users (NCEP, SPC)Special quality of service for key users (NCEP, SPC) Leverages existing infrastructure and serves as a Leverages existing infrastructure and serves as a
means for developing advanced networking means for developing advanced networking capabilitiescapabilities
Easy linkage with other networks (e.g., AWIPS, Easy linkage with other networks (e.g., AWIPS, NOAAPort)NOAAPort)
Could easily add other radarsCould easily add other radars Significant capacity for future growth (dual-pol, Significant capacity for future growth (dual-pol,
phased array)phased array)
The BenefitsThe Benefits
s fn
– Rapid and less expensive base data archival and Rapid and less expensive base data archival and retrieval at NCDCretrieval at NCDC Number of steps reduced to 2Number of steps reduced to 2 Archive time reduced by factor of 4Archive time reduced by factor of 4 Will improve archival rate (now 65%)Will improve archival rate (now 65%) Reduces data request turn-around timeReduces data request turn-around time Reduces data volume via compressionReduces data volume via compression Eliminates tape drive costsEliminates tape drive costs
– Possible secondary back-up/real time archives Possible secondary back-up/real time archives where advanced data perusal and web-based data where advanced data perusal and web-based data mining tools will be availablemining tools will be available
The BenefitsThe Benefits
s fn
Operational reliability and redundancy in Operational reliability and redundancy in a research networka research network
Network securityNetwork security The future of the I2/Abilene network The future of the I2/Abilene network
infrastructureinfrastructure Coordination with regional/state Coordination with regional/state
networksnetworks Capacity for growthCapacity for growth Phasing with deployment of the Open Phasing with deployment of the Open
RPGRPG
The ChallengesThe Challenges
s fn
Use of LDM could expand significantlyUse of LDM could expand significantly How to provide LDM support to private How to provide LDM support to private
sector users?sector users? Integration of LDM into NEXRAD Open Integration of LDM into NEXRAD Open
Systems Architecture (more later)Systems Architecture (more later) End-user visualization and analysis End-user visualization and analysis
software (not just a UCAR problem)software (not just a UCAR problem)
Challenges for UnidataChallenges for Unidata
s fn
CRAFT-2CRAFT-2 One of 4 National Abilene InitiativesOne of 4 National Abilene Initiatives Expand CRAFT-1 to additional 30-40 radarsExpand CRAFT-1 to additional 30-40 radars Possibly include FAA radarsPossibly include FAA radars OU and NCAR have an opportunity to OU and NCAR have an opportunity to
become joint online radar archive sites for become joint online radar archive sites for data mining, visualization, etcdata mining, visualization, etc
s fn
s fn
CRAFT-2 Radars Now Delivering CRAFT-2 Radars Now Delivering DataData
s fn
May soon begin turning offsome of the 8 mm recorders!!
WSR-88DWSR-88D
Abilene (T1Abilene (T1into NCDC)into NCDC)
NEXRAD Base DataNEXRAD Base Datafrom the 12 CRAFT Radarsfrom the 12 CRAFT Radars
LDM HDSS
MassMass Storage Storage
At NCDC TodayAt NCDC Today
s fn
– Web-based data miningWeb-based data mining– Analysis and visualization packagesAnalysis and visualization packages– Networking quality of service researchNetworking quality of service research– Digital libraries and distributed storage Digital libraries and distributed storage
technology researchtechnology research– Storm- and meso-scale data assimilation and Storm- and meso-scale data assimilation and
predictionprediction– Mesoscale process studiesMesoscale process studies– Small-scale climatologiesSmall-scale climatologies
OpportunitiesOpportunities
s fn
http://geosciences.ou.edu/~jsnow/Research/Meso/http://geosciences.ou.edu/~jsnow/Research/Meso/
MS Thesis by Brad Pugh (1999)MS Thesis by Brad Pugh (1999)Oklahoma Climatological SurveyOklahoma Climatological Survey
– NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC)NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC) $540K/3 years$540K/3 years Research ThrustsResearch Thrusts
– Test of direct ingest/archival at NCDCTest of direct ingest/archival at NCDC– Improve compression algorithmsImprove compression algorithms– Initial work on web-based data miningInitial work on web-based data mining
– NOAA earmark funding to OUNOAA earmark funding to OU $474K for 1 year$474K for 1 year Expand CRAFT to 30 radars (CRAFT-2)Expand CRAFT to 30 radars (CRAFT-2) Develop data assimilation capabilities for the WRF modelDevelop data assimilation capabilities for the WRF model Kelvin doing a mini-sabbatical at the NSSL this fallKelvin doing a mini-sabbatical at the NSSL this fall
– HPCC Proposal ($150K for 1 year, about to be funded)HPCC Proposal ($150K for 1 year, about to be funded) Data miningData mining Network quality of service researchNetwork quality of service research Hardware for additional radarsHardware for additional radars
Funding StatusFunding Status
s fn
The Future: FAA RadarsThe Future: FAA Radars Initial efforts are now underway to Initial efforts are now underway to
explore extending the CRAFT explore extending the CRAFT infrastructure to FAA radarsinfrastructure to FAA radars– TDWR (terminal Doppler weather radar)TDWR (terminal Doppler weather radar)– ASR (airport surveillance radar)ASR (airport surveillance radar)– ARSR (air route surveillance radar)ARSR (air route surveillance radar)
s fn
TDWRTDWR
s fn
Weber (2000)
Airport Surveillance Radars Airport Surveillance Radars (ASR-9)(ASR-9)
s fn
Weber (2000)
Airport Surveillance Radars Airport Surveillance Radars (ASR-11)(ASR-11)
s fn
Weber (2000)
Air Route Surveillance Radars Air Route Surveillance Radars (ARSR-4)(ARSR-4)
s fn
Weber (2000)
– NEXRAD is implementing an open-systems NEXRAD is implementing an open-systems software standard based on UNIX and software standard based on UNIX and other widely-used technology (e.g., TCP/IP)other widely-used technology (e.g., TCP/IP)
– The temporary RIDDS system becomes the The temporary RIDDS system becomes the Base Data Distribution System (BDDS)Base Data Distribution System (BDDS)
– Will require moving LDM and compression Will require moving LDM and compression into BDDSinto BDDS
Open RPGOpen RPG
s fn
Open RPGOpen RPG
s fn
Phase I: Current StrategyPhase I: Current Strategy
Repeater Hub
RIDDS Linux PC
Unidata LDM ($1500)
Internet Direct or 56K
Phone Lines
WSR-88D
At the radar site
Users
Internet2/Abilene
s fn
Phase II: BDDS + LDM/PCPhase II: BDDS + LDM/PC
Repeater Hub
BDDS Linux PC
Unidata LDM ($1500)
Internet Direct or 56K
Phone Lines
WSR-88D
At the radar site
Users
Internet2/Abilene
s fn
Phase III: Full BDDSPhase III: Full BDDS
Repeater Hub
BDDS + LDM
Internet Direct or 56K
Phone Lines
WSR-88D
At the radar site
Users
Internet2/Abilene
Efforts are now underway to create a national Efforts are now underway to create a national WSR-88D Level II (base) data ingest systemWSR-88D Level II (base) data ingest system
Real-time access for all users by subscribing to Real-time access for all users by subscribing to desired radars via LDM – LDM is a KEY component!desired radars via LDM – LDM is a KEY component!
ESDIM project and CRAFT-2 are investigating key ESDIM project and CRAFT-2 are investigating key issues (reliability, scalability, security)issues (reliability, scalability, security)
Hope to have an online digital library in Norman Hope to have an online digital library in Norman for data mining and research with another archive for data mining and research with another archive at NCARat NCAR
Pacing item for implementation is Open RPG/BDDS Pacing item for implementation is Open RPG/BDDS deployment (scheduled June ’01 – Dec ‘02)deployment (scheduled June ’01 – Dec ‘02)
New build for BDDS will be phased in New build for BDDS will be phased in simultaneouslysimultaneously
Tremendous opportunities for researchTremendous opportunities for research
SummarySummary
s fn
Top Related