Information Types and Registries
description
Transcript of Information Types and Registries
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Information Types and Registries
Giridhar ManepalliCorporation for National Research Initiatives
Strategies for Discovering Online DataBRDI Symposium – Feb 26, 2013
1
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Research Data Interoperability
2
Scientists, Data Curators,End Users, Applications
EnablingTechnologies
Discovery
Access
Interpretation
ReuseAccessed via Repositories
01000101..
ID
ID
ID
ID
ID
ID
ID
ID
ID
Datasets
01000101..
ID
ID
ID
ID
01000101..
ID
01000101..
ID
01000101..
IDID
ID
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Research Data Interoperability (cont.)
• Interoperability of research data allows discovery, access, interpretation, and reuse of datasets by researchers
• Examples• Discovery: A scientist from US “discovers” datasets from research
in Germany, in related or even unrelated domain• Reuse: A scientist from US “re-uses” or processes datasets from
the discovered research in Germany• For interpretation of accessible datasets, Types and Type
Registries play a significant role
3
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Information Types – Our Definition
• What they are not:• Programmatic data types (string, integer, double, etc.)• Mime types as normally used (text/xml, application/rdf)
• Types are identifiers that, with the help of associated metadata, characterize data structures used for managing information• Data structures could be at multiple levels of granularity
• Individual observations, to sets of observations within a time series, to multiple time-series sets that explain a phenomenon
• Usually • Spread across multiple files (each with specific mime type)• Distributed on the network (managed by various repositories)
• We call such data structures used for managing information digital objects
• Types (aka type identifiers) are unique across their user base• Types are associated with machine-readable metadata to support
interpretation of information• CNRI’s focus is to support infrastructure for enabling inter-discipline
types4
Type IDMachine Readable
Metadata
Digital Object
Network
Typed Digital Object
File File File File
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Value Proposition of Info. Types
• Typing allows • Grouping of digital objects generated in different
times and domains for reasoning and establishing correlations between different types of objects• Grouping is an aspect fundamental to humans for
reasoning about things• Creation of services that can automate
information processing based on information types
• Advanced information processing can be performed for finding unforeseen correlations, trends, etc. • This type of advanced processing has different
names: data-intensive science, fourth paradigm, big-data analytics, etc.
5
Type C
Type A Type B
Typed Digital ObjectCollection
Digital Object
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Value Proposition of Info. Types (cont.)
6
SUITE OF SERVICES
Visualization
I AgreeTerms:…
Rights
I AgreeTerms:…
I AgreeTerms:…
Data Set Dissemination1010011010101….
1010011010101….
1010011010101….
Data Processing
1. User requests Type from a Digital Object of interest.
1
5
2. Type ID is returned to the user.
Type Registry
Digital Objects
2
3 4
Interaction
3. User requests the Type Registry for the Type info.
4. Type Info is returned to the user containing Services Info.
5. User requests a Service for processing.
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Info. Typing Challenges• Challenge: When are two digital objects assigned the same type?• When the bit-level encoding matches?• Or when the higher-level structures and intent matches?• If two observations are made by two similar instruments at the same time on
the same entity, would the data generated by those two observations be constituted as being of the same type?
• Even if the data generated by each observation, similar in concept, has a different format (e.g., JPEG vs. PNG)?
• Our approach: • Intent wins over optics (formats, encodings, etc.)• The metadata associated with the type could list possible formats, encodings,
etc.• Alternative approach:• Establish a base type and then sub-type for accommodating variations• Our experience was that it was too cumbersome to deal with multiple
formats, encodings at the type definition level 7
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Info. Typing Challenges (cont.)
• Challenge: Can the same digital object be assigned multiple types?• If so, how do we deal with duplicate
types?• If not, how do we manage multiple
types assigned by several domains?• Our approach: • An object is assigned an inter-
discipline type only once. • Any domain-specific types are listed in
its metadata
8
Type α Type β
Typed Digital ObjectCollection
Type I
Machine-readableMetadata
Type αType β
Biologist Computer Scientist
Inter-discipline Type
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Info. Typing Challenges (cont.)• Challenge: How can existing information be typed under this
new scheme?• A lot of information exists already
• One approach: • Start with domain-specific types, if any, and generate domain-
neutral types and list the domain-specific types in their metadata records
9
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Info. Types – Machine-readable Metadata• Machine-readable metadata for Info. Types is still an area of research for us• Type interdependence• It is clear that sub-typing is needed for building on previously defined types• Our experience shows that sub-typing based on variations in formats and
encoding is a cumbersome process• Instead, an exhaustive list of possible formats and encodings may be specified
in the metadata• Domain-specific Types• Cross-domain Types could list or point at domain-specific types which could be
multiple for a given object, and which might define detailed semantics for interpretation
• Metadata for automated interpretation• For the few types of information we prototyped, defining metadata that helps
services process datasets is loose ended and sometimes impractical• A parsing-language or a pseudo-code may instead be captured that transforms
datasets into domain-specific ontologies or semantics10
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Info. Type Registries• Info. Type Registries are metadata registries that• Support recording of information types and associated metadata
records• Perform federation across other registries• De-duplicate (or match types) to control registration requests of
existing types• Include manual moderation and/or crowd sourcing function for
spotting redundant registrations (optional)• Cross-domain Type Registries may optionally link to domain-
specific Type Registries• Type Registries may manage or reference services that process
information of certain types• CNRI has vast experience building metadata registries 11
Corp
orati
on fo
r Nati
onal
Res
earc
h In
itiati
ves
Next Steps• Received Sloan Foundation funding to research Type Registries
within scientific and financial communities• CNRI employees lead and participate in a Type Registry
working group within the Research Data Alliance• Technical goal is to define the scope of ‘Information Type’ by
working in aforementioned projects, and build and release an open-source Type Registry in the next 18 months.
12