eReefs Data Brokering Layer
-
Upload
jonathan-yu -
Category
Data & Analytics
-
view
91 -
download
0
Transcript of eReefs Data Brokering Layer
eReefs Data Brokering Layer
LAND AND WATER FLAGSHIP | OCEANS AND ATMOSPHERE FLAGSHIP
Jonathan Yu | Research Software EngineerEnvironmental Information Infrastructures / IBMI / WRM / LWF 12 February 2014
Using semantics to support data discovery & access
Outline
1. Background/Use cases - eReefs
2. Data discovery/access challenges
3. Data Provider Node (DPN) concept and semantics
4. Data Brokering Layer (DBL) concept and API
5. Applications
eReefs What is the
health of the reef?
I want to know the Chlorophyll-a
concentration levels in the reef
Where are good diving sites to bring people?
Remote sensing
In-situ sensors(Weather stations
and monitoring sites)
Modelleddata
3 | eReefs Data Brokering Layer | Jonathan Yu
eReefs Data Brokering Layer | Jonathan Yu
Data provider nodes (DPN)
4 |
Feature Service
CoverageService
Map service
Geoserver MapInfo THREDDS 52north
Obsservice
?
API
Multiple combinations of services, interfaces, datasets for each DPN - relies on some cataloguing solutions
eReefs Data Brokering Layer | Jonathan Yu
Complex systems - inefficient, chaos
5 |
Portal X
Portal Y
eReefs Data Brokering Layer | Jonathan Yu
Current cataloguing solutions and challenges
Strengths• Harvest capability and keyword search• Integration with open data catalogs• Metadata standards (ISO19XXX and
others)
Limitations• Doesn’t quite capture DPN governance• Doesn’t resolve semantic heterogeneity• Doesn’t allow semantics-rich search
and discovery - limited to keywords
6 |
eReefs Data Brokering Layer | Jonathan Yu
Semantic Heterogeneity
7 |
Enviro Application
#1
Data
DB
Chl_MIM
Enviro Application
#2
Data
DB
mass_conc_chlorophyll_In_sea_water
Enviro Application
#3
Data
DB
mass_conc_chlorophyll_a_In_sea_water
eReefs Data Brokering Layer | Jonathan Yu
Semantic Heterogeneity… leads to data silos
8 |
Enviro Application
#1
Enviro Application
#2
Enviro Application
#3
Data Data Data
DB DBDB
Chl_MIMmass_conc_chlorophyll_In_sea_water
mass_conc_chlorophyll_a_In_sea_waterX X
Meetings
eReefs Data Brokering Layer | Jonathan Yu
Harmonised Publish, Discovery, Access and Use
9 |
Relies on community agreed vocabularies
Describe/Publish the data
Query/Use dataEnviro
Application #1
Enviro Application
#2
Enviro Application
#3
Data Data Data
DB DBDB
substanceOrTaxon= http://environment.data.gov.au/def/object/chlorophyll
scaledQuantityKind = http://environment.data.gov.au/def/property/chlorophyll_concentration
Need to communicate more consistentlyRequires shared, precise, agreed semantics
eReefs Data Brokering Layer | Jonathan Yu
eReefs Data Brokering Layer (DBL)
10 |
Data Brokering
Layer
End users
eReefsPortal
Remote sensing DPN
DPN Ontology
DomainVocabs
e.g. Water Quality
Hydrodynamic DPN
In-situ monitoring DPNCatalogue
APISCKAN,
pyCSW, ERDDAP
Flexible, ‘pluggable’ DPNs, data services and datasets
Flexible discovery and delivery of data
eReefs Data Brokering Layer | Jonathan Yu
DPN Semantic Description Example
11 |
dpn:Serviceorg:Organization dpn:Node
dpn:THREDDS
OC THREDDS Service
Ocean Colour DPN
CSIRO Ocean Colour Group
rdfs:subClassOf
rdf:type rdf:typerdf:type
DPN ontologyhttp://purl.org/dpn
DPN descriptioninstance
eReefs Data Brokering Layer | Jonathan Yu
Data provider nodes (DPN)
12 |
Feature Service
CoverageService
Map service
Geoserver MapInfo THREDDS 52north
Obsservice
DPN Service
Description
Lightweight RDF description
eReefs Data Brokering Layer | Jonathan Yu
DPN Semantic Description Example
13 |
dpn:Serviceorg:Organization dpn:Node
dpn:THREDDS
OC THREDDS Service
Ocean Colour DPN
CSIRO Ocean Colour Group
rdfs:subClassOf
rdf:type rdf:typerdf:type
THREDDS
NCI deployment
Data Brokering Layer
DPN registered with DBL
eReefs Data Brokering Layer | Jonathan Yu
Supporting domain vocabularies and def’s
14 |
http://environment.data.gov.au/water/quality/def/op
Observable Property ontology
eReefs Data Brokering Layer | Jonathan Yu
Example: Domain vocab term
15 |
eReefs Data Brokering Layer | Jonathan Yu
Example: Domain vocab term
16 |
... wqp:chlorophyll_a_concentration a skos:Concept, op:ScaledQuantityKind, qudt:ChemistryQuantityKind ;
skos:broader wqp:chlorophyll_concentration ; skos:prefLabel "chlorophyll a concentration"@en ;
eReefs Data Brokering Layer | Jonathan Yu
Searches over vocabularies
17 |
http://sissvoc.ereefs.info/search/?q=chlorophyll&endpoint=http://sissvoc.ereefs.info/sissvoc/ereefs
eReefs Data Brokering Layer | Jonathan Yu
Data annotated with bindings to vocab URIs
18 |
THREDDS
THREDDS Catalog
Domain Vocabs(Water Quality at
environment.data.gov.au)
Quantities/ Units ontology(QUDT)
substanceOrTaxon= http://environment.data.gov.au
/def/object/chlorophyll
scaledQuantityKind = http://environment.data.gov.au
/def/property/chlorophyll_concentration
unit= http://qudt.org/vocab/unit#Unitless
medium= http://environment.data.gov.au
/def/feature/ocean
eReefs Data Brokering Layer | Jonathan Yu
DBL Harvesting and End Use
19 |
Data Brokering Layer
THREDDS
THREDDS Catalog
Domain Vocabs(Water Quality at
environment.data.gov.au)
Quantities/ Units ontology(QUDT)
substanceOrTaxon= http://environment.data.gov.au
/def/object/chlorophyll
scaledQuantityKind = http://environment.data.gov.au
/def/property/chlorophyll_concentration
unit= http://qudt.org/vocab/unit#Unitless
medium= http://environment.data.gov.au
/def/feature/ocean
DPN Service
Description
End users
Client application
chlorophyll
eReefs Data Brokering Layer | Jonathan Yu
eReefs Visualisation Portal Prototype
http://vizportal.meteor.com
20 |
eReefs Data Brokering Layer | Jonathan Yu21 |
eReefs Data Brokering Layer | Jonathan Yu22 |
eReefs Data Brokering Layer | Jonathan Yu23 |
eReefs Data Brokering Layer | Jonathan Yu27 |
eReefs Data Brokering Layer | Jonathan Yu
Benefits
35 |
Data Provider X
Data Brokering
Layer
DPN Ontology
DomainVocabs
e.g. Water Quality
Catalogue APIS
CKAN, pyCSW, ERDDAP
End users
eReefsPortal
Data Provider Y
Flexible, ‘pluggable’ DPNs, data services and datasets
Allows flexible discovery and delivery of data
eReefs Data Brokering Layer | Jonathan Yu
Benefits
Decoupled system allows for expanding components in isolation:• DPN set• Vocabulary definitions
Enhanced data discovery
Aid operational monitoring of DPN services
Potential to facilitate data assimilation and automated data processing
36 |
eReefs Data Brokering Layer | Jonathan Yu
Future/Current Work
1. Test whether this the best approach for data services and dataset discovery- Compare/integrate with CKAN, Geonetwork, pyCSW, ERDDAP
2. Explore richer queries using semantics of the DPN metadata, vocabularies and services
3. Use DBL for data assimilation – hydrodynamic models with remote sensing
4. Demonstrate DBL applications beyond eReefs
37 |
eReefs Data Brokering Layer | Jonathan Yu
Summary
• The ‘Data Provider Node’ concept provides a way of thinking about data ownership, publication, implementation and governance
• Semantic web technologies allow for machine-readable DPN descriptions precisely and richly
• The ‘Data Brokering Layer’ concept uses the domain and DPN semantics to provide:
1. Flexible and ‘pluggable’ DPNs - data services and datasets
2. Allows flexible discovery and use of data
• Prototypes and demos via client applications
• Future work
38 |
LAND AND WATER
Thank you
Land and WaterJonathan YuResearch Software Engineert +61 3 9252 6440e [email protected]
Land and WaterBen Leighton
Software Engineert +61 3 9252 6516e [email protected]
Land and WaterShane Seaton
Software Engineert +61 2 6246 5825e [email protected]
More info:https://wiki.csiro.au/display/ER/Data+Brokering+Layerhttps://wiki.csiro.au/display/ER/Semantic+service+descriptionshttps://wiki.csiro.au/display/ER/Semantic+descriptions+of+datasets