UWG 2013 Meeting Publish-Subscribe (Datacasting).
12
UWG 2013 Meeting Publish-Subscribe (Datacasting)
-
Upload
bernard-parrish -
Category
Documents
-
view
217 -
download
0
Transcript of UWG 2013 Meeting Publish-Subscribe (Datacasting).
Goal PO.DAAC to publish availability of our data
Why? So users can subscribe and no longer have to
crawl FTP
Goal
High Level Functions: Publishing
Consumers
Archive
Ingest
Web Services & Publishing
Direct Data Access
High-Level Access Tools
Web Portal Visualization
Inventory
Information Providers
Data In Information In Data OutInformation Out
Data Providers
Datacasting Extension of RSS Work funded by NASA ACCESS 2005, 2009 Data providers publishes availability of new
data granules via XML feed Data consumers subscribes to feed to learn
about recently available data granules
Solution
Datacasting enables Down select items in feed based on multiple criteria
Temporal Spatial Other metadata
Automatically download selected data files Mash up across different types of feeds
Combine newspaper articles, scientific reports, event analyses, data, etc. in one place
And much more
By the way…
Publishing DatacastingRSS
Consumers
Archive
Ingest
Web Services & Publishing
Direct Data Access
High-Level Access Tools
Web Portal Visualization
Inventory
Information Providers
Data In Information In Data OutInformation Out
Data Providers
Data
cast
ingR
SS
DatacastingRSS<rss xmlns:datacasting="http://datacasting.jpl.nasa.gov/datacasting" xmlns:georss="http://www.georss.org/georss" xmlns:gml="http://www.opengis.net/gml" version="2.0"> <channel> <title>MetOp-A ASCAT Level 2 25.0 km Ocean Surface Wind Vectors</title> <link>http://podaac-test.jpl.nasa.gov/dataset/ASCAT-L2-25km</link> <description>This dataset contains operational near-real-time Level 2 ocean surface wind vector retrievals...</description> <pubDate>Wed, 27 Feb 2013 07:49:44 GMT</pubDate> ... <datacasting:channelUID>PODAAC-ASOP2-25X01</datacasting:channelUID> <datacasting:dataSource>METOP-A/ASCAT</datacasting:dataSource> <item> <title>ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc</title> <link>http://podaac-test.jpl.nasa.gov/dataset/ASCAT-L2-25km</link> <description>ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc</description> <enclosure length="847714" type="application/x-netcdf" url="ftp://podaac-ftp.jpl.nasa.gov/allData/ascat/preview/L2/25km/2013/057/ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc.gz" /> <pubDate>Wed, 27 Feb 2013 07:49:44 GMT</pubDate> ... <georss:where> <gml:Envelope> <gml:lowerCorner>-89.37 0.02</gml:lowerCorner> <gml:upperCorner>89.26 359.99</gml:upperCorner> </gml:Envelope> </georss:where> <datacasting:acquisitionStartDate>Tue, 26 Feb 2013 19:00:01 GMT</datacasting:acquisitionStartDate> <datacasting:acquisitionEndDate>Tue, 26 Feb 2013 20:41:59 GMT</datacasting:acquisitionEndDate> </item> ... <item> ... </item> </channel></rss>
Link to DatacastingRSS Feed
Client Tool: Scripts
Scripts in Python, Perl, MATLAB, etc. can be written to download data granules as they are made available
Sample MATLAB script for downloading data granules:xml = xmlread('http://podaac-test.jpl.nasa.gov/ws/metadata/granule/?format=datacasting&datasetId=PODAAC-ASOP2-25X01');
enclosures = xml.getElementsByTagName('enclosure');
for i = 0:enclosures.getLength-1 url = char(enclosures.item(i).getAttribute('url')); [pathstr, name, ext] = fileparts(url); filename = [name, ext]; if exist(filename, 'file') disp(['File exists: ' filename]); else urlwrite(url,filename); disp(['Downloaded: ' filename]); endend
exit;
Perl tool for subscribing to feeds and downloading and maintaining files on disk
Client Tool:Data::Downloader
% dado config init --filename ./ascat-l2-25km.conf% dado feeds refresh% dado files download
Terminal
Client Tool:Datacasting Feed Reader
Subscribe to DatacastingRSSFilter feed
items Download dataCreate mash up
