RANET GEONETCast Broadcast Manager: System Summary and Prospectus

8/14/2019 RANET GEONETCast Broadcast Manager: System Summary and Prospectus

1/16

August 2009, v1.00

RANET GEONETCast Broadcast Manager System Summary and Prospectus


2/16

This prospectus was prepared by the International Extension and Public Alert System (IEPAS) program and Program Manager at theUniversity Corporation for Atmospheric Research (UCAR) Joint Office of Science Support (JOSS) under award number NA06OAR4310119 from the US National Oceanic and Atmospheric Administration, U.S. Department of Commerce. Funding for theseactivities was provided by the USAID Office of US Foreign Disaster Assistance (OFDA) and the NOAA National Weather Service. Thestatements, findings, and recommendations are those of the author(s) and do not necessarily reflect the views of, or endorsement by,NOAA, USAID, UCAR, or any partners of either RANET or IEPAS.


3/16

Version Information / Update

August 3, 2009 (v1.00)- This document was prepared by Kelly Sponberg (Program Manager, IEPAS at UCAR JOSS). It provides anoverview of the RANET GEONETCast Broadcast Manager (RGBM), which includes basic functionality, use guidance, and details ongenerated products and data storage. The RANET GEONETCast Client Browser is described in brief detail. This document is meantas an introduction for content providers wishing to utilize the RGBM as a content management tool.

About RANET

RANET is a collaborative program of many National Hydro-Meteorological Services, non-government organizations, andcommunities. These varied partners come together to make weather, water, and climate information available to rural and remotepopulations, which are often most in need of environmental forecasts, warnings, and observations. The program works by advancingthe communication capabilities of national and regional organizations to reach rural and remote communities. It does so through avariety of activities that includes training, system/technology development, and demonstrations. Structurally RANET is owned andoperated at the national and community level. Basic system development and coordination is provided by several internationalpartners and designated leads. The US contribution to the RANET program is administered and coordinated by the InternationalExtension and Public Alert System (IEPAS) program at the University Corporation for Atmospheric Research (UCAR) Joint Office of Science Support (JOSS) under award number NA06OAR4310119 from the US National Oceanic and Atmospheric Administration(NOAA) U.S. Department of Commerce with funding from the USAID Office of US Foreign Disaster Assistance (OFDA).

Contact

Kelly Sponberg

Program Manager, IEPAS / RANETJoint Office of Science SupportUniversity Corporation for Atmospheric Research1850 Table Mesa DrBoulder, CO 80305

PH: +1-202-470-6022FX: +1-240-846-6123

http://www.ranetcommons.nethttp://geonetcast.ranet.ws


4/16

Summary Description

GEONETCast is a near real time, global network of satellite-based data dissemination systemsdesigned to distribute space-based, air-borne and in situ data, metadata and products to diversecommunities. 1

With roots in the meteorological and earth science community, the system isintended to support a variety of sectors whose information or applications involve an earthobservation component. These other communities could include agriculture, health, waterresource management, and disaster risk reduction to name but a few.

GEONETCast is a Task in the GEO Work Plan and is led by EUMETSAT, the United States, China,and the World Meteorological Organization (WMO). Many GEO Members and ParticipatingOrganizations contribute to this Task.

For nearly a decade RANET, in partnership with First Voice International (FVI), has operated twodata channels on the WorldSpace AfriStar and AsiaStar satellites. This older capacity isfunctionally similar to that provided by GEONETCast. Namely, both systems involve a satellitebroadcast covering continental scale regions. These are data broadcasts that allow for the

distribution of text, graphic, and multimedia formatted information to users with a compatiblesatellite ground station linked to a typical desktop computer. Both broadcasts are free to endusers, thereby limiting costs primarily to the initial equipment investment.

The application of these broadcasts supports a variety of users. The broadcast can be used asmeans to distribute core meteorological products to national weather services and warningauthorities. It can be used as a mechanism of national agencies to distribute information tointermediaries in rural locations, such as extension workers. And while more difficult tosustainably place, can even be considered for support of community information centers.

Based upon past use and success with the WorldSpace broadcasts, RANET has endeavored to

utilize the GEONETCast capacity in the Americas and Africa to support its ongoing programs andactivities.

The RANET GEONETCast Broadcast Manager(RGBM) is a web based tool for contentproviders to register and enter informationproducts into RANET capacity on theGEONETCast broadcasts. The goal of theRGBM is to provide a simple to use web formthat can be used to upload or create content,as well as command RANET servers toautomatically cull and update products fromweb locations, FTP servers, RSS feeds, and similar. With the RGBM, RANET has providedadditional API functionality, such that advanced users can create automation that pushesupdated content to the system as well. By providing this tool, the goal is to lower barriers to useof GEONETCast. Additionally, the RGBM indexes and categorizes information making it easier tofind and use by those receiving the broadcast.

1 http://earthobservations.org/geonetcast.shtml


5/16

The complimentary application to the RGBM is the RANET GEONETCast Broadcast Client. Whilenot covered in great detail in this document, the RGBC is a desktop application that utilizes alocal database to organize content received via the GEONETCast broadcast. Users of the clientcan opt to subscribe to various products, as well as customize their client interface to somedegree. While the client is the intended tool for managing downloaded broadcast material,users may alternatively use common desktop RSS readers, such as RSSOwl, to view and managecontent as well.

The following in this document provides an overview of the RGBM; outlining terminology,functionality, and how data and products are managed.

The RGBM is intended for content providers and managers. Those wishing to access and utilizethe system must e-mail the RANET Satellite Broadcast Service at [email protected].

Definitions and Basic Functionality of the RGBM

The RANET GEONETCast Broadcast Manager provides a simple to use form that exposes aconsiderable amount of advanced functionality to perform the following tasks:

1) Register content and define meta data on a specific product.2) Create automation that allows the RGBM to pull content from a network location, or

alternatively receive information pushed from a remote, and likely automated, source.3) Allow for manual upload and creation of content through the RGBM where automation

is not needed or possible.4) Formatting and broadcast of content entered into the RGBM in a variety of consumable

formats.

The RGBM organizes content into streams. Streams may be thought of as a single producttype. For instance an image of monthly observed precipitation anomalies for a particular regionmay be thought of as a stream. A product type, such as a cyclone watch for an ocean basin, mayalso be thought of as a stream. Similarly, a monthly periodical / newsletter would be considereda content stream.

Streams are really just a label for a relatively consistent product. The assumption is that theproduct is always updating, but streams could be used to register a static product as well; suchas a manual on water borne disease prevention. Another way to conceptualize streams is tothink of a web analogy. If you can link to a product, then you can create a stream. It does notmatter if this product is a bulletin, a feed of widely varying content, or simply an image.

Streams in the RGBM, however, are not the actual products, but rather the meta data thatprovides source information, product location / namespace, keywords, categories, and similarinformation. Individual products, the actual image or an issue of a periodical, are referred to aspayloads.

Payloads, as might be guessed, are associated with only one stream. In envisioning and buildingthe RGBM, payloads are also assumed to be culled through a variety of automated processes


6/16

identified when creating a stream. Nonetheless, payloads (specific products) can be manuallyuploaded to the system, or even created within the system through a basic WYSIWIG editor.

The distinction between stream and payload will become clearer when actually using thesystem.

Most other terms used in the RGBM will be self evident and defined in the system. This isparticularly true for categories, formats, and product type definitions required in the streamregistering process.

Two concepts / terms do require some upfront explanation. The RGBM, as noted before, iscapable of automatically updating content that is available on the internet. This may be from awebsite, news feed, FTP site, or other. Regardless, when registering a stream, the contentmanager can opt to tell the RGBM where it should look for updates new payloads. Within theRGBM this is referred to as auto-pull. This simply means that the RGBM can automatically pullupdates / payloads into the system for broadcast from a remote server resource.

Complimenting the auto-pull is the auto-push method. At times content providers may wish topush updated content to the RGBM; particularly for time sensitive material. This isaccomplished through a HTTP POST (REST) API, specially formatted e-mail, coded SMS (mobiletext) messages, and future functionality will allow for XML-RPC services. Regardless it isassumed that the content provider is creating her own automation to send the RGBM updatedcontent.

With auto-pull the RGBM periodically checks the content provider / source server for updates,whereas with auto-push, the RGBM waits for content to be sent to it by a remote server orother process.

Major Components of the RGBM

The RGBM is based upon Drupal 6.x, PHP, and MySQL; along with a number of custom Drupalmodules built specifically for the RGBM. This serves as the primary front-end for contentmanagers to register streams and potentially upload/create new payloads.

Backend processes to cull content, format payloads, and ultimately broadcast the content arebased upon a number of PHP and Python scripts. These are largely invisible to most users. Theexception to this is scripts designed to surface API functionality.

The Drupal frontend provides the following views and forms for content managers:

Stream Form (create / edit) Stream List (view / sort / discover) Payload Form (create / edit) System Health (failures, last updates, etc.) General RGBM Notices User Account Management (password, name, etc.)


7/16

Content managers are granted access with permissions that are specific to the individual.Individuals can then be associated with one or more institutions / content providers. When acontent stream is created, the creator must specify ownership based upon an institutional orcontent provider affiliation. Based upon this affiliation, any individual associated with thedesignated institution(s) may then modify the content stream, as well as manually upload /create payloads specific to the stream.

Broadcast Format and Data / Product Management in the RGBM

Content in the RGBM may come in a variety of formats. Most common image, text / document,and even audio and video formats are supported and allowed. (With multimedia formats,however, content providers should be aware that end users may not have the necessary playeror viewer to access the content.) As content is intended for a broadcast based system, productswith interactive dependencies to a server are prohibited. Namely AJAX or content relying onserver side scripts to function. Exceptions to this can be made for niche applications where

users are expected to possess and internet connection or where the content can be retrofittedto operate off a local database and server system. At this time executables, batch files, scripts(.py, .vbs, etc.) and similar are also prohibited, unless providing the program / script in advanceto RANET for review, scanning, and testing. While not prohibited, self-contained interactivecontent, such as Flash, AIR, and similar will be audited once entered into the system. Thiscontent will be blocked from broadcast until audited, which may take upwards of 72 hours. Allother content, including basic text and images are periodically audited. The RGBM alsomonitors payloads for deviations in size and format. At certain thresholds significant changeswill result in an audit.

To prevent the same content (payload) from registering in the system multiple times, content

that is gathered via auto-pull and auto-push is compared with previous entries via an MD5 filehash. If a file with the same hash exists, the newly pulled or pushed files will be discarded.

Once content is successfully captured by the RGBM through auto-pull, auto-push, or manualmethods, the RGBM creates a number of wrappers for the content. Original file names are notpreserved, but rather translated into the following:

__.

The following wrappers are created for each content stream:

RSS 2.0 ATOM (Common Alerting Protocol [CAP] modification) RGBM-XML (specific XML feed for RGBM) HTML (a basic, flat HTML formatted list with local/relative links to payloads)

Additionally meta-data and log files are also generated. These files have no appreciable functionin either the RGBM or the RGBC (client browser), but rather are generated for tracking potentialerrors, providing hash information to ensure proper download of the broadcast, etc.


8/16

The content manager defines the number of payloads (individual products) to list as items in thevarious XML based feeds and HTML list. This is done when registering a content stream. Forinstance the content manager may choose to list only the five most recent satellite images. Therange for retaining content can be set between 1 and 50 items. The default is 10.

When broadcast, all of the above mentioned files are contained in a compressed ZIP file. Thisfile decompresses the files to a folder identified by the stream registry ID. This ID is a simplenumber. All content associated with a stream, as well as the wrappers and log files, will becontained in this folder. A master index and registry of all streams is also broadcast, whichprovides guidance to the RGBC and other viewers as to the location and description of content.The master index is also human readable; unlike the RGBM file naming conventions andwrappers.

Viewing Broadcasted RGBM Content

While not covered in detail in this document, the counterpart to the RGBM is the RANET

GEONETCast Broadcast Client Browser (RGBC). This viewer is an Adobe AIR based applicationutilizing a SQLite database. Functionally, it is an XML driven browser that can render HTML,text, and common image formats such as .gif, .jpg, and .png. The Client can also display somemultimedia formats (e.g.- Flash), and unsupported file types are simply launched from the RGBC.

The RGBC serves to manage downloaded content by uncompressing the ZIP files to a localinstallation directory. It will also serve to delete files / content that is outdated or marked fordeletion by the user. Where the RGBC is not utilized, a simply Python script is provided for thisbasic data management function.

Once content is downloaded and uncompressed, the RGBC indexes the payloads and streams in

its local database. Users can then use the GUI to search for content streams by keyword,source, title, categories, etc. To help users easily access content of interest, they may alsosubscribe to content. In actuality all content is downloaded via the broadcast. Subscribedcontent simply provides a shorter list of content the user has selected and will view on a regularbasis. The subscription view can be set to sound a visual and audible cue when new content in asubscribed stream has been downloaded. Future versions of the Client will allow e-mail andSMS notifications based upon subscriptions as well.

Finally, the RGBC collects data on content views, subscriptions, as well as tracking of successfuldownloads. This information is anonymously reported back to the RGBM if users possess aninternet connection. Only the aggregate statistics are collected and are not referenced by user,IP, geolocation, or other identifiable information. Users of the RGBC, however, may opt out of this reporting. The reporting feature is intended to provide RANET with statistics on products of interest and use.

While the RGBC is the intended method of viewing and accessing content from the RGBM, usersmay use a local RSS reader application to view content. A master list of all streams is providedin the master download directory as: rgbm_all_streams.html and rgbm_all_streams.rss. Usersmay manually copy the RSS links and feeds listed in this index into the reader. The reader will


9/16

then pull the RSS 2.0 wrapper to view and access content. Not all readers allow for local feeds,and many provide different functionality.

All feeds are also posted on the RGBM website for users wishing to receive data by both internetand GEONETCast.

Finally, users may simply access content through the rgbm_all_stream.html file by following theHTML links in this index. This is perhaps the most cumbersome way to navigate content, but forusers focused on a few specific products may be the desired method.

Of course users can create their own readers and viewers utilizing any of the XML based feedsprovided in by the RGBM.

Accessing the RGBM

Content providers should log into the system at http://geonetcast.ranet.ws. 2

To receive log in

credentials, potential users should contact [email protected]. Please note, however, RANET mayrequire proof of institutional affiliation before granting access. This may require an exchange of letters and potentially contacting your organization head.

If a potential user is not affiliated with a government institution (international, national, orlocal), RANET may require a description of content to be sent on the system. Commercialinformation, advertisements, and similar solicitations are strictly prohibited. Content promotinga religious and political view is also not allowed. And it should go without saying thatinflammatory, prejudiced, or hateful content is also prohibited.

RANET reserves the absolute right in all circumstances to determine what to prohibit in terms of

content, individuals, and institutional sources of information.

To be granted access to the RGBM, aside from proof of institutional affiliation, potential userswill be required to provide a valid e-mail address, mailing address, and phone number. Thesewill not be shared or utilized in any way except for basic system notifications.

If a user account is suspected of being compromised / hacked, it will be immediately suspendeduntil the owner can be validated. If suspended, all content streams associated with the user willalso be suspended until their integrity is verified.

Creating a Content Stream in the RGBM

Users should click on the Create Stream link to register a new product stream in the RGBM.This will bring up an admittedly long form. While long, it is relatively straight forward, and it isonly required once to establish a stream. Users should note that many of the fields arecollapsed in certain areas. Simply click the area title to expand the form.

2 Once the beta period for the RGBM has ended, the system will operate over a SSL connection. As aresult this web location will change to https://geonetcast.ranet.ws.


10/16

For all text fields you may enter information in any language you wish, assuming the charactersare supported by UTF-8. While the forms are currently limited to English (this will change innear future versions), the product can be published in any language appropriate to the end user.

Begin creating a stream by providing the Stream Name. This should generally be the producttitle. For a meteorological product, it might simply be something like GHA Seasonal ClimateOutlook. Try to be as descriptive as possible, while also using plain language. Avoid simplystating something like 10-Day Forecast, as there may be many 10-day forecasts. Provide eitherthe relevant area of the forecast or the name of the institution creating the product.

Next provide an abstract or description of the product. This should outline what it is, perhapsprovide guidance on use of the product, as well as describe update frequency. If the product isa periodical magazine, describe the topics covered and intended audience.

Now provide Keywords. These should be separated by commas and can constitute a singleword, acronym, or phrase. For users utilizing the RGBC viewer, keywords support the productsearch functionality. You should include as many appropriate terms as possible and eveninclude source provider name. Try to think of synonyms, and as possible provide terms inmultiple languages appropriate for the intended audience. For instance if the product is a

Satellite Rainfall Estimate for the Mekong River Basin, include the term precipitation as well asrainfall.

The Stream Name and Description fields are completed. Now click on the source informationlink to expand this area.


11/16

Begin this area by providing the acronym or other short name of the source / institutionproviding the information. In the following field provide the full formal name of the institution.

Next, provide the institutions namespace. This should generally be the main website. It shouldnot be the link to the specific product.

If a contact (e-mail) for the source can be provided, please do so. This source should be thecontact responsible for generating the specific product stream. This information will only beused in case there are technically difficulties updating content, such as a change in web locationor if a product is discontinued.

Now open the Meta Data area.


12/16


13/16

By default the system will assume you are not using auto-pull functionality. If you wish to do so,select Yes. From the pull down menu, select the network protocol and format type. Thesemay be FTP and a variety of HTTP resources, such as an RSS feed, an image file, etc.

Next, if using the auto-pull functionality, you must provide the URI of the content and locationwhere it should be culled. Generally this is a simple HTTP URL. For password protectedlocations, follow the format of http://:@domain.com/directory/file.ext.

Now open the Stream Serial and Platform Selection Area.


14/16

The Stream Serial is a unique ID assigned to your content stream. You need not and should notalter this field. It is provided as a reference. The serial is used for auto-push methods. If usingthe auto-push HTTP API method, you must provide one or more public facing static IP addresses.Only API requests using the stream serial and matching the specified IP will be accepted.

Finally, select which broadcast platform will be utilized to deliver your content. You may selectboth, however, if your content is region specific, then RANET requests limiting distribution toonly the relevant region / broadcast.

Now open the Time Frequency and Broadcast Suspension area.


15/16

The stream update frequency is largely used only by the auto-pull methods. Select thefrequency in which your content generally updates. RANET servers will poll your servers asappropriate. Note that if the content stream updates weekly, do not select a high frequencyupdate schedule. This will only needlessly cause the RANET servers to waste network capacity.The system will poll more frequently than described in the pull-down menu, and it will learnover time when content is generally released. For instance initially when you select a onceweekly update, the system will poll your server twice daily. Over a few months, if the systemnotices your product is updated reliably every Monday, it will focus its polling on that day.

The next pull down menu allows you to specify how many payloads / updates to keep in theindex and various XML feeds. The default is ten, but you may select a range of anywhere from 1 50. This is should largely reflect the update frequency of your content. Content that isupdated hourly should likely retain more items so that the end user is provided with a day ortwo of information and data.

Finally, you can opt to suspend a stream. In the RGBM it is not possible to ever delete either acontent stream or payload; except by the system administrator. Both may be suspended,however, which will remove them from the broadcast, as well as all indexes. After 6 months, if astream or payload has remained suspended, it will be permanently removed from the system.Note that if you suspend a stream, users will still be able to access previously broadcast content

until it is locally deleted on their system. A notice that the stream is suspended, however, willbe provided.

Now open the Stream Use and Edit Permissions area.


16/16

RANET GEONETCast Broadcast Manager: System Summary and Prospectus

Documents

Transcript of RANET GEONETCast Broadcast Manager: System Summary and Prospectus