CM [A] R’s “MarLIN” Metadata System - or, how do we discover what data we’ve got??

www.csiro.au

CM[A]R’s “MarLIN” Metadata System- or, how do we discover what data we’ve got??

Tony Rees

Manager, Divisional Data Centre

3 June 2005

CSIRO Marine Research

CMR’s “MarLIN” Metadata System

Talk overview

Data in our overall activities Data directories and metadata CMR’s “MarLIN” system Creating metadata content – roles and

issues Concluding remarks Questions / discussion items

MarLIN ... Marine Laboratories Information Network (1998)

... Marine & Atmospheric Research Laboratories Information Network (2005 onwards)

1


The research process (heavily simplified!)

Problem Formulation

Operationalisation

Data Acquistion

Data Appraisal

Data Interpretation

Solution / New Knowledge

2



Problem Formulation

Operationalisation

Data Acquistion

Data Appraisal

Data Interpretation


Scientific publications, reports, products

Happy customers

New problems

outputs:

2



Problem Formulation

Operationalisation

Data Acquistion

Data Appraisal

Data Interpretation


Data Sources

Data Sinks

Scientific publications, reports, products

Happy customers

New problems

outputs:

2


Data sources and sinks

Data sources

pre-existing data in CMR / CSIRO holdings

pre-existing data – external (third party) sources

new data collection / generation

3



Data sources




Data sinks

project / researcher archives (formal / informal)

CMR centralised data holdings (e.g. in Data Centre)

external repositories / customers’ holdings

3



Data sources




Data sinks

project / researcher archives (formal / informal)

CMR centralised data holdings (e.g. in Data Centre)

external repositories / customers’ holdings

feed back to...

3


Need for a data catalogue

Requirements

(1) Need to know what we already have... across our Division across whole agency

(2) Others may also need to know what we have (or subset of the same, that we are in a position to share)

(3) Also need to know what exists elsewhere (potential for acquisition of third party data, where it exists)

4


Need for a data catalogue

Requirements

(1) Need to know what we already have... across our Division across whole agency

(2) Others may also need to know what we have (or subset of the same, that we are in a position to share)

(3) Also need to know what exists elsewhere (potential for acquisition of third party data, where it exists)

Solutions

(1 and 2) Catalogue of our own data assets, using “metadata”

(3) Other agencies’ catalogues, metadata gateways

plus other routes (literature searches, peer-to-peer networking, etc.)

4


What is this “metadata”?

“Metadata is information about data or other information.” (USGS web site)

“Metadata is data about data. In other words, it is a structured summary of information that describes the data. Metadata includes, but is not restricted to, characteristics such as the content, quality, currency, access and availability of the data.”

(ANZLIC Metadata Guidelines v2, 2001)

5


What is this “metadata”?

“Metadata is information about data or other information.” (USGS web site)

“Metadata is data about data. In other words, it is a structured summary of information that describes the data. Metadata includes, but is not restricted to, characteristics such as the content, quality, currency, access and availability of the data.”

(ANZLIC Metadata Guidelines v2, 2001)

In practice ...

Metadata is structured summary information about [......]

E.g.:

a film guide holds metadata about films e.g. title, director, genre, cast, running length, story synopsis, language, rating...

a scientific publications database holds metadata about scientific publications e.g. author, title, location in journal, publication date, abstract, keywords...

a data directory (metadata system, in the present context) holds metadata about datasets of value to scientists and other potential users.

5


Why metadata?

Structured information collection supports powerful information retrieval

Efficient – easier to search / browse the metadata than obtaining and interrogating all the actual resources, in the first instance (i.e., metadata is a surrogate for the resource)

Metadata is human-readable and text searchable, resource may not be (e.g. rock specimens, images, music, digital data files...)

Collection of metadata into metadata systems supports resource discovery (entry point/s for information)

Captures “corporate memory” – essential information required to understand or re-use the resource (information does not solely reside in people’s heads)

Assists in resource management – knowing what one has is a precursor to managing it well

Can be used for resource distribution – enquirer locates the metadata, then is provided with an access point to the data (e.g. with a web-based system, can then hyperlink to any web-accessible data source).

6


Who uses metadata?

Science agencies and jurisdictions use it to describe their data holdings – e.g. (in our sphere of interest):

Australian Antarctic Division Australian Hydrographic Service BRS Bureau of Meteorology Dept. of Environment and Heritage [EA] Geoscience Australia...

Jurisdictional directories: ACT, NSW, NT, SA, TAS, VIC, WA

Overseas examples – some agency-based, some jurisdictional, some national/thematic (e.g. European marine data, international “global change” data, space/satellite data, etc.)

Frequently, metadata “push” is coming predominantly from the “spatial data” community, i.e. data with a geographic component – but similar principles can be applied to (virtually) any data.

7


Example: the Australian Spatial Data Directory (ASDD)

Single gateway (portal) to search 20+ metadata systems around Australia concurrently

CMR is represented (but no other CSIRO Divisions currently have a metadata system)

Searching is fairly basic, individual system entry points often have more functionality (e.g. MarLIN), however both have their place.

8


How it works in practice...

online data files / databases documents (digital and

non-digital)offline data archives

images (graphics, photos, video)

specimen collections

www user (internal / external)

CMR metadata system

describes / points to ...

metadata gateways, search engines

(e.g. ASDD, Google, etc.)

(etc.)9


An overseas metadata example...

10

The UK’s “National Biodiversity Network” (www.nbn.org.uk) holds datasets on species distribution surveys for UK birds, animals, invertebrates, plants, etc...



11



12



13


In other words, metadata supports...

Dataset discovery – via lists and / or structured searches

Dataset appraisal (via descriptive information) – including what (dataset content) where (dataset spatial footprint) – if applicable when (dataset temporal footprint) who by, why, etc.

Dataset access constraints – who can access, under what conditions

Dataset location and access point

Supplementary information e.g. documentation, images, references, etc.

14


Metadata standards

Metadata example just shown – own format (no externally recognised standard)

Standards assist interoperability – e.g.

USA – 2 standards currently (1 small one “DIF”, one large one “FGDC”)

UK / Europe – historically, little standardization – but new ISO standard (2003) now exists (based on US “FGDC” model)

Australian uses “ANZLIC” standard (v.2) – currently pre-ISO, next version will be ISO compatible.

15


What’s in the ANZLIC standard?

16



Dataset title, ANZLIC identifier

Custodian organisation and contact

Abstract, search words, bounding box, and Geographic Extent Name or polygon

Start / end dates, progress and maintenance status

Access constraints, stored and available data formats

Data quality (lineage, positional & attribute accuracy, completeness, and logical consistency)

Metadata entry (or last update) date

“Additional metadata” (for anything else)

16



Dataset title, ANZLIC identifier

Custodian organisation and contact

Abstract, search words, bounding box, and Geographic Extent Name or polygon

Start / end dates, progress and maintenance status

Access constraints, stored and available data formats

Data quality (lineage, positional & attribute accuracy, completeness, and logical consistency)

Metadata entry (or last update) date

“Additional metadata” (for anything else)

Note:

- Bounding box, and start / end dates support spatial and temporal searching

- “Search words” support structured searches and information retrieval

- Remaining fields searchable as free text.

16


What’s missing from the ANZLIC standard (but would be useful)?

17


What’s missing from the ANZLIC standard (but would be useful)?

Originator organisation (for data obtained from elsewhere)

Contributors, Acknowledgements, References

“Global Project” affiliation (e.g. WOCE, JGOFS, etc.)

Better keywords (ANZLIC ones are very high level)

Better geographic footprint (e.g. by grid squares or similar) – especially for patchy / irregular sampling patterns

CMR Project affiliation

Voyage or survey name (and relevant details)

Species names (if relevant)

Data volume and attributes in the dataset, plus information about its local storage environment

Links to documentation, graphics, and the data itself (where available)

(probably some other stuff too, but that is a good start).

17


CMR metadata standard = “extended ANZLIC”...

ANZLIC + useful extras

= “CMR metadata standard”(1998 onwards)

- informal set of elements of value to our operations

- also, prototype for draft CSIRO metadata standard (2002).

18


The external context – Australian Government

19



Commonwealth Statement – via Office of Spatial Data Management (OSDM)

extract from: AUSTRALIAN GOVERNMENT CUSTODIANSHIP GUIDELINES [for spatial data in this instance]: The Rights and Responsibilities of Spatial Data Custodians

“1. Various Australian Government agencies hold large amounts of spatial data, and will continue to collect more in the future. To achieve efficient and effective acquisition, management and use of spatial data, custodian agencies will be given policy guidelines setting out custodianship rights and responsibilities.

19






... 32. A key part of any set of spatial data is the accompanying metadata [...] The custodian of the data is normally the best placed to supply this information.

19






... 32. A key part of any set of spatial data is the accompanying metadata [...] The custodian of the data is normally the best placed to supply this information.

... 34. The custodian is expected to facilitate efficient and effective use of the government's data, so as to derive maximum benefit from the investment. Thus the metadata must always be readily available, not just for existing users, but for potential users. The custodian should maintain publicised points of contact for enquiries and be in a position to provide the appropriate metadata promptly.”

19


The external context – CSIRO

20



extract from: [draft] CSIRO Scientific Data Management Policy – as submitted to Executive, March 2002 [technically still a “draft awaiting approval”]

under Scientific Data Management Roles, Responsibilities and Actions:

Corporate

“Senior management (CEO, Deputy CEOs, Business Unit Chief Executives) are to foster and encourage the development of a culture within CSIRO where:-

the value of scientific data and associated data management is recognised and rewarded;

scientific data assets are shared by staff within the Organisation, and where appropriate, with others outside the Organisation.”

20



extract from: [draft] CSIRO Scientific Data Management Policy – as submitted to Executive, March 2002 [technically still a “draft awaiting approval”]


Corporate

“Senior management (CEO, Deputy CEOs, Business Unit Chief Executives) are to foster and encourage the development of a culture within CSIRO where:-

the value of scientific data and associated data management is recognised and rewarded;

scientific data assets are shared by staff within the Organisation, and where appropriate, with others outside the Organisation.”

Research projects

“The incorporation of scientific data management objectives into routine R&D planning and development procedures. This should include procedures that will ensure the recording and updating of metadata, ensure the current and future security of the data asset (backup and archiving), ensure the protection of CSIRO’s intellectual property, and resolve issues of data ownership and future access.”

20



extract from: CSIRO Scientific Data Management Policy – as submitted to Executive, March 2002 [technically still a “draft awaiting approval”]


Individual Officers

“Adopt a ‘one-CSIRO’ view of the data they collect, analyse, back-up and archive;

Adopt and utilise the CSIRO Metadata Standard and make the recording of metadata a routine part of their work practices; and

Ensure the security of CSIRO scientific data assets.”

(Comment:)

... the above are mainly “sticks”, “carrots” will be discussed later in the presentation.

21


What is a “dataset”, in this context?

22



According to the ISO metadata standard (ISO 19115, 2003): “Dataset: an identifiable collection of data.(NOTE - A dataset may be a smaller grouping of data which, though limited by

some constraint such as spatial extent or feature type, is located physically within a larger dataset [...] A hardcopy map or chart may be considered a dataset.)”

22



According to the ISO metadata standard (ISO 19115, 2003): “Dataset: an identifiable collection of data.(NOTE - A dataset may be a smaller grouping of data which, though limited by

some constraint such as spatial extent or feature type, is located physically within a larger dataset [...] A hardcopy map or chart may be considered a dataset.)”

“In practice” definition ... a collection of data sharing common features such as data type, data collection activity or data assembly purpose, management / availability as a discrete unit, etc.

Size of data “chunks” to be described (aka dataset granularity) is a subjective choice – whatever best suits the data custodian, or is most valuable to prospective data users

Basically it comes down to a “lumping” or “splitting” decision (or set of guidelines) – however splitting down to the atomic level is probably undesirable in this context, for practical considerations.

22


The CMR metadata story so far...

CMR has the “MarLIN” metadata system – implemented 1997-8 (plus ongoing enhancements) – in-house software, based on EA original (we can modify further as needed)

Records can be “internal”, = CSIRO only (for confidential or third party data), or “public” (open access)

Currently holds 2,100+ dataset descriptions – c.1,000 of these describe centrally-held datasets (Data Centre holdings)

Coverage of other Divisional holdings is patchy at present (a few groups have made an effort, many have not)

To address this in part, it is proposed to construct “skeleton” (template) records for all Divisional science projects (CMR plus CAR, i.e. future CMAR) – to act as a starting point for projects to describe their data holdings

Also, management / project “buy in” to the concept needs further developing.

23


Searchability

Structured searches – including browse by subject categories, keywords, projects, custodian (site), voyages, species names, and more

Free text searches

Lists of titles – including recent additions / updates

Search by space and time criteria

Search for “own” records via Edit interface

Search via ASDD (Australia-wide metadata gateway) – public records only

Also – text searches via “Google” etc. will find relevant public records.

24


Example: search (browse) by taxonomic group

25


Example: search (browse) by taxonomic group


Free text search... (e.g. for specialist terms, person names, etc.)

26


Free text search... (e.g. for specialist terms, person names, etc.)

27


Full metadata record (start)

(etc.)28


Full metadata record (start)

(etc.)

Includes grid square representation for spatial “footprint” of datasets (where possible)

28


Present MarLIN Edit interface (metadata entry tool) - start

(etc.)

29


Where to find MarLIN

30


Sample MarLIN usage – 6 months to 11/5/2005

CMR accesses (searches + page views) – 945 / month

other Australian accesses (non CMR): 6,000 / month

international, + “origin unavailable” accesses: 10,000 / month

(NB excludes search engine hits – over 50,000 / week)

... presumably, some people arrive here by accident, others are “just looking”, however a percentage are genuinely after CMR data!

Also, process works the other way – persons requesting data from our holdings can be pointed to MarLIN for the relevant metadata.

31


Creating MarLIN Content – roles?

Data Centre staff

describe Data Centre holdings (centrally managed data)

maintain the metadata system, provide user assistance, etc.

may undertake rescue and description of legacy data (where resources available)

maintain currency of Data Centre’s metadata records

32



Data Centre staff





Project data custodians

describe their own data holdings (including data acquired from third parties)

maintain currency of their own metadata records

may pass data to central holdings for archiving / online access – with appropriate metadata

32



Data Centre staff





Project data custodians

describe their own data holdings (including data acquired from third parties)

maintain currency of their own metadata records

may pass data to central holdings for archiving / online access – with appropriate metadata

Project / RG leaders, Divisional management

promote / facilitate / monitor project-level metadata activity?

32


Recap - what are we trying to do here? (activities)

Describe our data holdings – to the inside and outside world

Bring together relevant dataset documentation (or pointers to it) in a single, www-accessible location includes incentive to source, digitise, and web-

enable relevant images, text documents, item-level lists, etc.

Provide a tailored set of search tools which suit our data holdings and audience

Provide access to our metadata (and data where appropriate) – on a self serve, 24/7 basis

Connect our entered information to the wider world for “discovery” purposes, e.g. to metadata gateways and internet search engines.

33


Benefits of good metadata to the Organisation (impacts)

Assists in data management – knowing what we have is a precursor to managing it well

Assists our researchers to do their job better – plan projects, locate data quickly, assess gaps, reduce duplication of effort / multiple purchases of third party data

(Should be particularly valuable in multi-site, multi-Division environment such as CSIRO – counteract intentional or unintentional “silo” effect)

Facilitates information and data access, for internal and external users (e.g. self service via the web, not reliant on human responses)

Communicates / promotes our resources to the wider world, demonstrates compliance with relevant policies / legislation

Auto-generation of relevant data listings – for reporting purposes, etc.

Captures “corporate memory” – key aspects about data necessary for its current and future discovery, appraisal, and re-use.

34


Concluding remarks

MarLIN content – a “work in progress” (continually moving target...) – some runs on the board, still a way to go...

35


Concluding remarks


Potential “trailblazer” / proof-of-concept for a one-CSIRO system (one day)

but still waiting for “buy in” at corporate level to good data management principles and tools

35


Concluding remarks




Good vehicle for promotion / reporting / dissemination of information about Divisional research activities (in tandem with communications, scientific publications, etc.), if that is what we want

35


Concluding remarks




Good vehicle for promotion / reporting / dissemination of information about Divisional research activities (in tandem with communications, scientific publications, etc.), if that is what we want

Compliance issues seem to have a back seat at present, may become more significant in future, e.g. ...

obligations on public-funded agencies to describe their data holdings

obligations on completed projects to archive and describe their data along with final reports (etc.)

35


NB, Data Centre undertakes a range of activities, MarLIN is only one of them...

36


“Data Centre Space”

Divisional Data Warehouse

Long term project data stores (“data islands”)

Historic + non-digital project data

“MarLIN” metadata system (data

catalogue)

“CMR Project Space”

OBIS

AODC-JF (2005-6)

Oceans Portal (2005-6)

ASDD (metadata)

“Data Trawler”(data access)

National Facility Research Vessel

Manned data requests service

External Automated Systems

Digital project data – finite duration

other DC systems (including off line

archives)Data

Centre holdings

DATA SOURCES

DATA USERS

Client Groups:• CMR Project Staff• CMR Managers• National Facility Users• External Collaborators• General Public

export to / access via ...

Project dataadvice /

assistance(3 CMR sites)

CAAB c-squares

- master species dictionary- species distributions- etc.

- spatial search and retrieval- online mapping

MarLIN as a component of Data Centre activities

37


Are we on the right track here (at all?)... are we currently doing enough / too much / not enough in the metadata

arena?

... does the system fulfil current user (client) expectations – e.g. Divisional staff users, others?

... how do we determine “metadata usefulness”?

Some questions / discussion points

38



arena?



Who are the “business owners” (beneficiaries) of the concept? Whose problem is it if metadata does not get created? (current / future)


38



arena?




What is the situation in other CSIRO Divisions?


38



arena?





How does the upcoming CMR / CAR merger impact on future metadata activities?


38


Are we on the right track here (at all?)... are we currently doing enough / too much / not enough in the metadata arena?





How does the upcoming CMR / CAR merger impact on future metadata activities?

What are the resource implications (i.e., costs) of maintaining accurate and current system content?

e.g. AAD model – 1 person [1 x FTE] to oversee system content, nag assist metadata creators, etc. – for xx science staff


38

www.csiro.au

Thank You

To visit MarLIN:

go to CMR home page (www.marine.csiro.au/)

>> Data Centre (www.marine.csiro.au/datacentre/)

>> MarLIN (www.marine.csiro.au/marlin/)

Please contact Data Centre staff at any site for further assistance / information.

Contact CSIRO

Phone: 1300 363 400+61 3 9545 2176Email: [email protected]: www.csiro.au


(supplementary slides)


Potential barriers / obstacles to be overcome

Lack of incentives (carrots, i.e. rewards, recognition) and / or obligations (sticks) for projects / staff to create metadata

Perceived lack of usefulness (incomplete coverage, information can be out of date) – researchers use other more “reliable” methods to locate the data they need

Describing own data is a low-level chore, other tasks have higher priority (or are more interesting, or produce more immediate benefits)

Some people do not wish to share “their” data anyway, describing it in a publicly accessible system is counter-productive

A lot of data is not in a re-useable state (e.g. not digitised, incomplete, disorganised, not documented), lack of resources to address this

Metadata entry tool is cumbersome, and asks too many “hard” questions about the data

Once created, metadata then has ongoing maintenance overhead associated with it (never ending task...)

39


Possible solutions...

Lack of carrots / sticks

Perceived lack of usefulness (incomplete coverage)

Low priority task

Resistance to sharing “own” data

Poor data state

Metadata entry tool complicated

Ongoing metadata maintenance requirement

40


Possible solutions...

Lack of carrots / sticks

Perceived lack of usefulness (incomplete coverage)

Low priority task

Resistance to sharing “own” data

Poor data state

Metadata entry tool complicated

Ongoing metadata maintenance requirement

Cultural change – including top-down policy Integration into project workflow

Increased participation; “skeleton” record for every project

as (1) above; maybe with regular audit / reporting function

Cultural change

Better data management practices Targeted resources, for priority legacy datasets?

Revamp / upgrade metadata entry tool; user instruction / assistance

as (1) above; also some Data Centre involvement

40


online and offline resources

metadata repository

describes / points to ...

metadata entry tool (edit interface)

content creator (internal)

www user (internal / external)

metadata search tool

content moderator

metadata gateways, search

engines

remote browse / search capability

re-edit ...

Components of the metadata system

database administrator

41


sneak preview of “upgraded” MarLIN Edit interface (under construction, May ’05)

42


future agency system

metadata systems

describe / point to ...

CMR

MarLIN

CMR data

DEH

EDD

DEH data

BoM

BoM data

GA

GA data

etc.

etc.

National Metadata Infrastructure

43


future agency system

metadata systems

describe / point to ...

CMR

MarLIN

CMR data

DEH

EDD

DEH data

BoM

BoM data

GA

GA data

etc.

etc.

ASDDAustralian Spatial Data Directory – national

cross-agency metadata gateway

search via ASDD – search across multiple agencies, basic functionality

search via MarLIN – search only CMR holdings, but extra functionality (also view “CMR internal” records not visible to external users)

National Metadata Infrastructure

43

CM [A] R’s “MarLIN” Metadata System - or, how do we discover what data we’ve got??

Documents

Transcript of CM [A] R’s “MarLIN” Metadata System - or, how do we discover what data we’ve got??