CEOS WGISS 24 - IDN Task Team

Task leader: Lola Olsen, represented by Tyler Stevens

Agenda1. GCMD/IDN Science User Working Group2. Review of Minutes and Action Items3. Update on Population and Usage Statistics4. Development Plans for MD9.8 - MD105. GEOSS Activities and other Collaborations6. Node Reports7. Evolution of Controlled Vocabularies8. Re-emergence of the Interop and Communication9. The CEOS IDN Newsletter

Briefing on the GCMD IDN Science User Working Group

Meeting, June 19-20 2007 by Dr. Wyn

Cudlip

Key Topics from WGISS 23Key Topics from WGISS 23

New IDN Features in MD 9.7New IDN Features in MD 9.7 Evolution of the DIFEvolution of the DIF Future DevelopmentFuture Development Information System and Data Center

(ISDC) for Geo-scientific Data User Working Group (UWG) MeetingUser Working Group (UWG) Meeting IDN Contributions to IDN Contributions to

Interoperability HandbookInteroperability Handbook

WGISS Action ItemsWGISS Action Items

No. 22-13: When JAXA is ready, add a link to JAXA multilingual prototype to the GCMD page. No. 23-8: Arturo to report back to WGISS-All on the awareness of the GEO task teams of the new IDN portals that have been created for those GEO tasks, and clarify what reviews are currently underway for the new GEO portals.

Status: 23 June 2007: Lola and Arturo distributed a memo describing the portals that are currently in place, and they have been register in GEO registry. CLOSED

No. 23-10: Arturo (IDN Team) to assess the differences between the platform and instrument descriptions in the IDN and those contained in the CEOS database (CEOS Earth Observation Handbook).

No. 23-22: The IDN team (Lola Olsen) to perform a detailed analysis of user query patterns and present a summary of usage patterns and lessons learned at WGISS-24.

Characteristics of IDN UsersCharacteristics of IDN Usersthrough Population and Usage through Population and Usage

Statistics Statistics

Most Popular Keyword Search

December 2005 - October 2007

DIF Population by Earth Science Topic

Most Popular Full-Text Searches

December 2005 - October 2007

Searches by Ancillary Keywords

Platforms

12%Instruments

14%

DIF Population Growth

Deletion and eventual

replacementof Antarctic DIFs

DIF Population by IDN Node

New, Revised, and Deleted DIFs - 2007

Quality Control is on-going

Problem with Google Cache

Web Page Hits

Summer Declines

Users by Continent

North America

Europe

Asia

Australia

South America

Africa

Other

January 2007 - October 2007

Users by European Country

Google (50%)

GCMD Direct Access (20%)

Yahoo

(6%)

NA

SA

(4%

)

Others (20%)

wikipedia.orgnoaa.govusgs.gov

The IDN allows robots to crawl the site to help broaden user access.

How the Users Access the IDN

When Users Access the IDN

Lets us know best time to schedule server

maintenance.

January 2007 - October 2007

Most Popular Browsers and Platforms

Important for cross-platform and browser

support of IDN software.January 2007 - October 2007

Metadata Contributions January-September 2007

From docBUILDER45%

55 %

Improved translation technology has increased contributions from sources using other “mandated” standards.

Lessons Learned from User Stats

Concentrate resources on keyword interface Concentrate resources on keyword interface because most consistent usage come from there.because most consistent usage come from there.

Consider adding spell-corrector for full-text Consider adding spell-corrector for full-text search to give users more chances of finding search to give users more chances of finding data.data.

Improve functionality of docBUILDER and Improve functionality of docBUILDER and metadata ingestion tools.metadata ingestion tools.

Capitalize on Google cache and provide additional Capitalize on Google cache and provide additional mechanisms for accessing metadata (metadata in mechanisms for accessing metadata (metadata in KML).KML).

Use semantic web technology to offer the option Use semantic web technology to offer the option to search on related words to their original query.to search on related words to their original query.

Development Plans forDevelopment Plans forMD 9.8-MD 10MD 9.8-MD 10

MD 9.8 - Migrate from MD 9.8 - Migrate from Oracle to “Open Source” Oracle to “Open Source”

Software (MySQL)Software (MySQL)

Staff evaluated alternative databases: Staff evaluated alternative databases: • PostgreSQL, Mckoi, MySQL, and XML PostgreSQL, Mckoi, MySQL, and XML

databases.databases.

MySQL chosen as alternative to Oracle.MySQL chosen as alternative to Oracle.

Staff is now in the process of migrating to Staff is now in the process of migrating to MySQL.MySQL.

Oracle: 4610 ms. MySQL: 1582 ms.

(# of milliseconds to query 150 Atmospheric Keywords)

Oracle Vs. MySQL Database

•GCMD Oracle: 4610 ms.•GCMD XML DB: 6361 ms.•OS X MySQL: 1582 ms.•OS X XML DB: 2978 ms.•GCMD BDB (Cache): 2378 ms.•OS X BDB (Cache): 1589 ms.

(# of milliseconds to query 150 Atmospheric

Keywords)

Benchmark Report

MD 9.9 - Quality UpgradeMD 9.9 - Quality Upgrade

To improve user experience To improve user experience To enhance interoperabilityTo enhance interoperability To reduce maintenanceTo reduce maintenance To streamline software testing processTo streamline software testing process

MD 10 - Next Generation IDN

Empower data providers to manage Empower data providers to manage their “own” metadata (load, query, their “own” metadata (load, query, share, etc.) through a distributed share, etc.) through a distributed input capability.input capability.

Represent the GCMD vocabulary as Represent the GCMD vocabulary as an ontology.an ontology.

Re-engineer server to support rising Re-engineer server to support rising levels of data sets and increasing levels of data sets and increasing demands for services.demands for services.

GEOSS Activities and Other GEOSS Activities and Other Activities/CollaborationActivities/Collaboration

IDN Registered in GEOSS IDN Registered in GEOSS Registry Publication PortalRegistry Publication Portal

GEOSS PortalsGEOSS Portals

IDN Architecture Within GEOSS

Import metadata into GEOSS Portal

cataloguesMetadata content harvest

from FGDC clearinghouse

node

IDN

IDN Components in

GEOSS registry

IPY Data & Services Portals

Find IPY DIFs in

New Portals Added Since May 2007

NASA DataNASA Data THREDDSTHREDDS OBIS-SEAMAPOBIS-SEAMAP EBA-AMD SCAR ProjectEBA-AMD SCAR Project GEOSS WildfiresGEOSS Wildfires GEOSS Wildfires GEOSS Wildfires

ServicesServices GEOSS InSitu GEOSS InSitu

InstrumentsInstruments European Space AgencyEuropean Space Agency Essential Climate Essential Climate

VariablesVariables

The “King” Request

Mapped IDN Keywords to

Essential Climate

Variables (ECV)

Refine by Platforms Under

Ozone To Get Detailed Platform

Description

Enhanced Platform & Instrument Descriptions

Expanded database table structures for Instrument and Platform

hierarchical keywords

10 Platform Categories66 Platform Series/Entities

3 Instrument Categories25 Instrument_Classes29 Instrument_Types 22 Instrument_Subtypes

Requirements for a Requirements for a Platform/Instrument Search Platform/Instrument Search

Interface Interface

Capitalize on existing platform and instrument Capitalize on existing platform and instrument ancillary information. ancillary information.

CEOS Earth Observation HandbookCEOS Earth Observation Handbook NASA Science Reference HandbookNASA Science Reference Handbook IDN DescriptionsIDN Descriptions

Utilize new technology to create an intuitive Utilize new technology to create an intuitive interface.interface.

Provide capability to sort platform and instrument Provide capability to sort platform and instrument information by columns in sequential order.information by columns in sequential order.

Enhance capability to refine search within interface.Enhance capability to refine search within interface. Directly link from instrument to platform Directly link from instrument to platform

descriptions and vice-versa, and link directly to descriptions and vice-versa, and link directly to associated metadata.associated metadata.

Left Hand Navigation

NASA Earth Observation Platforms

Launch Date

TERRA Polar Sun-Synchronous 1999-12-18 TERRA METADATA

UARS ACRIMCLAESHALOEHRDIIMLSPEM

SAMSSOLSTICE

SUSIMWINDII

Polar Non-Sun-

Synchronous 1991-09-12 UARS METADATA

ASTERCERESMISR

MODISMOPITT

Platform Name Orbit Type Associated Instruments Metadata

MOCK UP

Left Hand Navigation

NASA Earth Observation Instruments

MODIS AQUATERRA

2002-06-25 MODIS METADATA

IIR CALIPSO IIR METADATA

Instrument Name Associated Platforms MetadataStart Date

Passive

Instrument Class

Passive 2006-04-28

MOCK UP

docBUILDER Tool for Platforms - docBUILDER Tool for Platforms - DemoDemo

Fields for Platform Description in GCMD

Fields for Platform Descriptions in CEOS Earth Observation Handbook

Platform IDPlatform ID Mission (Short Name)Mission (Short Name)

Platform IdentificationPlatform Identification

(Associated to GCMD controlled (Associated to GCMD controlled keywords )keywords )

+ Platform Category+ Platform Category

+ Platform Series/Entity+ Platform Series/Entity

+ Platform Short Name+ Platform Short Name

+ Platform Long Name+ Platform Long Name

Mission (Long name) Mission (Long name)

ApplicationsApplications

Platform DescriptionPlatform Description

(Summary)(Summary)ApplicationsApplications

Platform LogisticsPlatform Logistics

+ Launch date+ Launch date

+ Launch Site ( Pull down list + Launch Site ( Pull down list database)database)

+ Estimated Design Life (time)+ Estimated Design Life (time)

+ Primary Sponsor ( option to add + Primary Sponsor ( option to add various sponsors )various sponsors )

+ Launch Date+ Launch Date

+EOL : End of Life+EOL : End of Life

PLATFORMS COMPARATIVE TABLE IN RESPONSE TO TASK WGISS 23-10

Note: Description information provided in the document task were created by Olsen, L.M., G. Major, S. Ritz, T. Stevens, M. Meaux, A. Restrepo, 2007 . NASA/Global Change Master Directory (GCMD) Earth Science Keywords.

Fields for Platform Descriptions in GCMD

Fields for Platform Descriptions in CEOS Earth Observation Handbook

Online ResourceOnline Resource(URL field provided)(URL field provided)

URLURL

OrbitOrbit+ Orbit Type ( Pull down list menu to + Orbit Type ( Pull down list menu to choose orbit type)choose orbit type)+ Orbit Altitude ( km)+ Orbit Altitude ( km)+ Orbit Inclination (degrees)+ Orbit Inclination (degrees)+ Equator Crossing (Local time )+ Equator Crossing (Local time )+ Period (minutes)+ Period (minutes)+ Repeat Cycle (days)+ Repeat Cycle (days)+ Perigee (km)+ Perigee (km)+ Apogee (km) + Apogee (km)

Orbit Details and URLOrbit Details and URL+ Type+ Type

+ Altitude+ Altitude+ Inclination+ Inclination

+ Period+ Period+ Repeat Cycle+ Repeat Cycle

Platform-based InstrumentsPlatform-based Instruments InstrumentsInstruments

Synonymous Platform NamesSynonymous Platform Names ( Short ( Short names, Acronyms, Abbreviations of names, Acronyms, Abbreviations of alternative names)alternative names)

Fields for Platform Descriptions in GCMD

Fields for Platform Descriptions in CEOS Earth Observation Handbook

Creation DateCreation Date( Date this description was created)( Date this description was created)

Revision DateRevision Date (Future date to review (Future date to review this description)this description)

Sample ImageSample Image( To provide URL with an image of the ( To provide URL with an image of the platform) platform)

IDN NodeIDN Node( Platform owner or Agency sponsor)( Platform owner or Agency sponsor)

Platform LogisticsPlatform Logistics

+ Launch Date+ Launch Date

+ Design Life+ Design LifeStatusStatus (i.e., Operational, Proposed, (i.e., Operational, Proposed, Being developed )Being developed )

INSTRUMENTS COMPARATIVE TABLE IN RESPONSE TO TASK COMPARATIVE TABLE IN RESPONSE TO TASK WGISS 23-10WGISS 23-10Note: Description information provided in the document task were created by Note: Description information provided in the document task were created by Olsen, L.M., Olsen, L.M., G. Major, S. Ritz, T. Stevens, M. Meaux, A. Restrepo, 2007 . NASA/Global Change Master G. Major, S. Ritz, T. Stevens, M. Meaux, A. Restrepo, 2007 . NASA/Global Change Master Directory (GCMD) Earth Science Keywords.Directory (GCMD) Earth Science Keywords.

Fields for InstrumentsFields for InstrumentsDescriptions in GCMDDescriptions in GCMD

Fields for InstrumentsDescriptions in CEOS EarthObservation Handbook

Instrument IDInstrument ID Instruments (Short Name)Instruments (Short Name)

Instrument IdentificationInstrument Identification(Associated to GCMD controlled (Associated to GCMD controlled

keywords)keywords)+ Instrument Category+ Instrument Category+ Instrument Class+ Instrument Class+ Instrument Type+ Instrument Type+ Instrument Subtype+ Instrument Subtype

Instruments (Long name)Instruments (Long name)Measurements and ApplicationsMeasurements and Applications

TypeType

Instrument DescriptionInstrument Description

Integrated Sensors Integrated Sensors (Associated to GCMD controlled (Associated to GCMD controlled

keywords)keywords)

Associated PlatformsAssociated Platforms(Associated to GCMD controlled (Associated to GCMD controlled

keywords)keywords)

MissionsMissions

Spectral/Frequency DomainSpectral/Frequency Domain+ Wavelength Keyword+ Wavelength Keyword+ Number Channels+ Number Channels+ Spectral/Frequency+ Spectral/Frequency+ Coverage/Range+ Coverage/Range+ Spectral/Frequency Resolution+ Spectral/Frequency Resolution

Technical CharacteristicsTechnical Characteristics Waveband (included channels)Waveband (included channels) Swath widthSwath widthAccuracyAccuracy

Fields for InstrumentsDescriptions in GCMD

Fields for InstrumentsDescriptions in CEOS EarthObservation Handbook

Instrument LogisticsInstrument Logistics+ Data Rate+ Data Rate

(bps/kbps/Mbps/Gbps/Pbps)(bps/kbps/Mbps/Gbps/Pbps)+ Instrument Start Date+ Instrument Start Date+ Instrument Stop Date+ Instrument Stop Date+ Instrument Owner+ Instrument Owner AgencyAgency

Online ResourceOnline Resource

Sample ImageSample Image

Creation DateCreation Date

Revision Date info Revision Date info

IDN Node (IDN Node (Platform owner or Agency Platform owner or Agency sponsor)sponsor)

AgencyAgency

Instrument Start DateInstrument Start Date

Instrument Stop DateInstrument Stop DateStatusStatus

Follow-up-up

Evaluate the addition of fields within the IDN Instruments description.

Swath width Accuracy

Continued addition of Instruments and Platforms from CEOS Handbook.

Online Demo to add Platforms and Instruments descriptions – previously viewed.

Suggestions and questions from WGISS-24 participants.

Node ReportsNode Reports

Bernd Ritschel - GFZ ISDC

AMDAMD

AMD Search Page Hits

Over 10,000 Hits in 2005

~ 14, 000 Hits in 2006

~ 40 % increase in 2006 from 2005

AMD Search Page Hits vs. Retrievals

Over 5,500 Retrievals in 2005Over 6,500 Retrievals in 2006

Over 10,000 Retrievals in 2007 (Jan - Jul.)

Launching of IPY 2007-2008?(over 500 unique IPs)

XXIX SCAR/JCADM Meeting

Note: Stats shown only represents 55% of total portal stats from July 2006 – August 2007

Hits by Country

Average visitors per day 19Total repeat visitors 1,235Most frequent domain: AustraliaVisit site between 9pm-10pmHighest volume day of week: Tuesday

Evolution of Controlled Vocabularies

Controlled Keywords Lists List is dynamic: keywords may be changed, deleted, or added.

Changes are implemented according to the Keyword Rules http://gcmd.gsfc.nasa.gov/Resources/valids/rules.html

In addition to Science Keywords, the GCMD maintains the following controlled keyword lists:

Services Keywords, Data Resolution, Locations, Instruments, Wavelength Keyword, Platforms, Orbit Type, Launch Site, Projects, Data Centers, URL Content Types, Chronostratigraphic Units (geologic time), IDN Nodes

Controlled Keywords Recent/Upcoming Changes

Keyword hierarchy extended to 5-Levels to assist in refinement of the search.

Recent Changes:• Atmospheric Chemistry

• Biological Classification (New Topic)• Ecological Dynamics (Revised)

•Terrestrial Hydrosphere (Renamed)

Upcoming Changes:• Atmosphere

• Human Dimensions• Oceans

• Biosphere

NEW

Atmospheric Science Keyword Revision (February - October 2007)

“The Long Journey to the GOLDMIND: Steps Towards Developing a Comprehensive Keyword Ontology”

2. Do the Research

3. Collaborate

4. Persevere

The GOLDMIND

1. Set a Goal

1. Set a Goal

A Complete Revision of the IDN Atmospheric Science Keyword list in 2008

Science Keyword Revision

“The Long Journey to the GOLDMIND”

2. Do the Research

• Review DIFs in the GCMD

• AMS Glossary Online

• Online publications through “Google Scholar”

• Consult data providers and experts

• Review Text Books:

• Critchfield, “General Climatology”

• Holton, “An Introduction to Dynamic Meteorology”

• Wallace/Hobbs, “Atmospheric Science: An Introductory Survey”

Science Keyword Revision

“The Long Journey to the GOLDMIND”

3. Collaborate

• Meet with project manager - Lola Olsen

• Consult Data Providers and Experts

Science Keyword Revision

“The Long Journey to the GOLDMIND”

4. Persevere

• Be willing to accept small steps towards a larger goal.

• Be patient when a trail leads to a roadblock and cheerfully backtrack and try a new trail.

• Never give up!

Science Keyword Revision

“The Long Journey to the GOLDMIND”

The Journey is almost over! The GOLDMIND is in sight.

Science Keyword Revision

“The Long Journey to the GOLDMIND”

GCMD Ontology Language for Discovery of Metadata INformation and Data

Reemergence of the Interop Reemergence of the Interop and Communicationsand Communications

Need for Communication

With increased usage, we must With increased usage, we must communicate:communicate: DIF/SERF modifications DIF/SERF modifications Keyword changes Keyword changes Software changes Software changes User Support/FAQ User Support/FAQ

Technologies to help with Communication:Technologies to help with Communication: Mailing ListsMailing Lists ForumsForums WikisWikis Content Management Systems (CMS)Content Management Systems (CMS) RSSRSS

CEOS IDN Standards Interoperability Forum

(INTEROP)

Moving to ceos-idn-interop@lists.nasa.gov

Continue to use GNU MailmanContinue as a moderated list

Possible future split of list:ceos-idn-interop (for general questions/discussions)

ceos-idn-keywords (for keyword announcements)

Communication Tools: Mailing Lists

Communication Tools: RSS Feeds

The IDN RSS feed allows users to view the most current Data Set and Service descriptions added to the IDN. Once a data set or service description is added, it will show up in your RSS feed.

Communication Tools: Communication Tools: Forums/Bulletin BoardsForums/Bulletin Boards

Communication Tools: Wikis

Communication Tools: Content Management

Systems/Collaboration

Communication Tools:Cyber

Infrastructure/Community

One Final Communication…. Final Communication….

The IDN/GCMD Strategic Plan

Thank You

Tyler.B.Stevens@nasa.govGIS/Services Coordinator

Lola.M.Olsen@nasa.govTask Leader

Special Thanks to the GCMD stafffor their assistance and contributions.