Metadata & Interoperability

ENVE/CE 424/524

Metadata

• They describe the entire contents of a data set– metadata are data about data– the documentation and handling instructions for data

• Metadata are what make data useful– without documentation and handling instructions data would have no

value to a user– without metadata it would be impossible to find data in a library or on the

WWW

What does metadata tell you?

"Using a standard to create metadata enables the GIS community to speak in like terms. It also sets the basis for software development by standardizing items and item descriptions. This has enabled the development of cataloging and indexing tools as well as search engines which can pose queries against collections of metadata."

Why use metadata?

•We all create metadata informally, as notes or reports

•We all use data that we didn't create

•When you're trying to use someone else's data, which is easier to interpret, informal notes, a voluminous report, or a formal standard? Use the history of word-processing as an example. It used to be that you never knew what software would be on any given computer... Wordstar, WriteNow, WordPerfect, Word... Maybe you got familiar with using your favorite word processor, but if you had to use another person's machine, there'd be no guarantee that the software you were familiar with was on it. Gradually, word-processing software evolved to the point that different word-processing programs applied the same standards for creating and formatting, and furthermore most programs can automatically read and open a document created in a different format. Establishing standards for word-processing has made it possible to access and use documents regardless of specific formats, programs, or platforms.

•Like any powerful tool, the metadata standard requires some time and effort to learn, but it is a worthwhile investment in the long run.

What does metadata allow you to do?

• Automate the process of searching for data– compare using a library catalog

• Determine the fitness of a data set for use– particularly regarding quality

• Handle data effectively– e.g., format

• Identify notable data contents– e.g., to find images of an interesting hurricane

Metadata can be Expensive to Generate

• They represent a high level of abstraction– and may need an expert to define

• How do you reach consensus?– I say road you say street.

• But the benefits are substantial– metadata make it possible to find data sets, and use them

effectively– they allow the benefits of investments in data to be realized

The U.S. FGDC Standard

• To be useful, metadata must adhere to widely accepted standards

• Most widely used standard is the US Federal Geographic Data Committee’s Content Standards for Digital Geospatial Metadata (CSDGM)

• Defined by a committee of U.S. Federal agencies

• Now widely used worldwide

• The basis of a new international standard

• Potentially several hundred items for one data set– but easily boiled down to a much smaller number

• Defines the items that should be included in a data description but not how they should be formatted or structured

Federal Geographic Data Standard

FGDC Metadata Example

The Dublin Core Standard

• Devised by the digital library community

• Suitable for any type of data, geospatial included– easily extended to include essential items for geospatial data

– e.g., the latitude and longitude limits of the data set’s coverage

• The idea is to establish a minimum set of metadata elements that define the fundamental properties of a dataset.

Elements of Dublin Core

TitleAuthor or CreatorSubject or KeywordsDescriptionPublisherOther ContributorsDateResource TypeFormatResource IdentifierSourceLanguageRelationCoverageRights Management

International Organization for Standardization (ISO)

• TC 211 - Standardization in the field of digital geographic information.

• Aims to establish a structured set of standards for information concerning objects or phenomena that are directly or indirectly associated with a location relative to the Earth.

• These standards may specify, for geographic information, methods, tools and services for data management (including definition and description), acquiring, processing, analyzing, accessing, presenting and transferring such data in digital/electronic form between different users, systems and locations.

• The work shall link to appropriate standards for information technology and data where possible, and provide a framework for the development of sector-specific applications using geographic data.

•ISO and FGDC work together to homogenize their standards.

eXtensible Markup Language (XML)The use of XML provides a reference framework for encoding of nested data structures and a means to test them.

Unlike HTML, which mixes content tags such as <title> and <body> with presentation tags such as bold <b> and italics <i>, XML provides strictly for the structure of information entities using beginning and ending tags.

Tags can be user defined

User communities define their standard tags and apply them to the structure of metadata standards.

ArcGIS Metadata

Geolibraries

• Repositories of data that can be searched for data covering geographic areas of interest

– this was very difficult in a conventional library using a card catalog

– each data set in a geolibrary is identified with a geographical footprint

– in a search, footprints are matched to the area of interest defined by the user

The Alexandria Digital Library

The user picks an area of interest by interacting with a map, specifying latitude and longitude

limits, or giving a place name. The library

returns all data sets whose footprints match

the query area, and which match other

criteria also supplied by the user.

National Spatial Data Infrastructure (NSDI)

• ‘the technology, policies, standards, and human resources necessary to acquire, process, store, distribute, and improve utilization of geospatial data’

Source: Presidential Executive Order #12906 (1994): 'Co-ordinating Geographic Data Acquisition and Access: The National Spatial Data Infrastructure' W. Clinton.

1) Establish standards2) Improve access to and sharing of data3) Develop national digital geospatial data framework

The vision of the NSDI: to assure that spatial data from multiple sources (Federal, State, and local governments, academia, and the private sector) are widely available and easily integrated to enhance knowledge and understanding of our physical and cultural world.

National partnerships via NSDIs• The problem:

– Data duplication commonplace – so waste occurs

– Ad hoc data sharing has many difficulties

– Data often tailored to one application

– Best data often collected in greatest detail at local level but not accessible to regional or national folk

– Indexes/metadata to available geospatial data unknown until recently

– No general protocols for any of this until NSDI.

The solution involves:Federal government (many agencies)State governmentLocal governmentPrivate sector – contractors, value-adders, exploitersNot for profit organizationsPublic

A global partnership – the need

‘In any area of the world, a number of core data types are required to assess environmental conditions and develop strategies that can lead to long term sustainable human development.

For many areas of the world, these data do not exist at the scales required to support these activities. Development of such data sets is labor-intensive and costly and, although these data sets could support a wide variety of applications, no single use can generally justify the full cost of development.’

Htun (1997) UN Regional Director for Asia

Governments and the private sectors

• National governments own and control national mapping agencies

• All such mapping produced to national specifications until recently

• New private sector providers:– Produce imagery for anywhere in world

– Produce road databases

• How do we get these to work together?

A Global Spatial Data Infrastructure?

• Difficult enough to get national players to work together…

• Is GSDI a process, a general framework or a product?

• Who are the stakeholders?

• Who needs it? (military doing what they need themselves?

The Global Mapping Project

http://www.iscgm.org/html4/index.html

Interoperability

Parallels the push for open systems - the intention to adhere to vendor-neutral computing standards

Interoperability has a range of meanings but all focus on the ability to move easily from one system to another.

What is required:

• Data producers must ensure that their data are readily accessible and understandable

• User must be able to identify and locate relevant information

• Queries to dispersed sites must be formulated in a manner meaningful on both data user and provider ends

• Geodata from one source must be capable of being integrated with data from another

• Display and analytical functions must available

The Open GIS Consortium (OGC)

• The Open GIS Consortium (OGC) is a not-for-profit, international consortium whose 250+ industry, government, and university members work to make geographic information an integral part of information systems of all kinds.

• Operates a Specification Development Program that is similar to other Industry consortia (W3C, ISO, etc.).

• Also operates an Interoperability Program (IP), a global, innovative, partnership-driven, hands-on engineering and testing program designed to deliver proven specifications into the Specification Development Program.

OGC VisionOGC Vision

A world in which A world in which everyone benefits everyone benefits

fromfromgeographic geographic

information and information and services made services made

available available across any network, across any network,

application, or application, or platform.platform.

OGC MissionOGC Mission

To deliverTo deliverspatial interface spatial interface

specificationsspecificationsthat are openly that are openly

available for global available for global use.use.

Geospatial One-Stop

Geospatial One-Stop makes it easier, faster, and less expensive for all levels of government and the public to access geospatial information. The Geospatial One-stop is one of 24 e-government initiatives sponsored by the Federal Office of Management and Budget to enhance government efficiency and improve citizen services.

• E-Government (E-gov) Initiative– Component of President Bush’s Management Agenda – One of 24 E-Government initiatives– Part of the Government to Government Portfolio

• Vision: the geographic component for e-gov

• Accelerate implementation of the National Spatial Data Infrastructure (NSDI)

GeoSpatial One Stop Tasks

1. Establish integrated Framework Data Content Standards

2. Use metadata to inventory, document and publish Framework data holding in NSDI Clearinghouse

3. Publish metadata for planned data acquisition and update for Framework data

4. Prototype and deploy enhanced data access and web mapping services for Federal Framework data

5. Establish a Federal Portal as a logical extension to the NSDI Clearinghouse Network

• human request handling

• ftp data access

• web discovery and order

• documentation of local content models

• use of proprietary formats

• little adherence to enforceable standards

• data translation largely the responsibility of the customer

Providers ofTheme X

customer

• automated ad-hoc requests• data and mapping via web services• web discovery, portrayal, and order or access• support of common content models and extended models• support of neutral encoding methods• data packages can be validated against a common reference• data translation shared between provider and customer

Providers ofTheme X

middlewarecommon packaging

services

customer

Data services Past and Future

Geospatial One-Stop Portal Concept

• Portal is an online access point for geospatial data, maps of data, and metadata about data

• Distributed Information: data stored at provider sites, not in Portal

• Portal uses open standards for interoperability– Geospatial web service specifications

– Metadata and content standards

• Portal builds upon NSDI Clearinghouse by providing direct access to data – Web-based user interface for human access

– Application interface for automated GIS/DSS access

Future Distributed Information System

Local Governments

The National Map

State Governments

Tribal Governments

Other Federal Agency

DoD

Commercial Sector

Local Users

DoD Users

Tribal Users

State Users

National PolicyMakers

Commercial Users

Civilian Users

International Users

Federal Users

NSDIOne-Stop

Portal

replicated access points

Services

Data Bases

Article Review

1. Give full citation of the article (also URL if online) If not online, provide a copy of the article.

2. Include an article summary - a long-worded rewrite is not necessary. Make sure you hit the main points. What was the take-home point of the article?

3. Compare the article to analysis methods presented in class. Did the authors deviate or use a more advanced method. What were their assumptions? Why did they or didn’t they use “standard” methods discussed in class and lab?

4. State what you learned from the article/chapter. How did this help you with your project? With understanding concepts covered in lab or lecture?