Distributed Databases and Applications Presentation

DiGIR 1

Distributed Databases and Applications

John Wieczorek

Museum of Vertebrate Zoology, UC Berkeley

DiGIR 2

Distributed Databases – Discipline-specific

The Species Analyst (TSA) The Integrated Taxonomic Information System (ITIS) FishNet The Mammal Networked Information System (MaNIS) HerpNET The Ornithological Information System (ORNIS) …

DiGIR 3

Distributed Databases –International

European Natural History Science Information Network (ENHSIN)

Biological Collection Access for Europe (BioCASE)

Australia Virtual Herbarium (AVH) Red Mundial de Información Sobre

Biodiversidad, Comisión Nacional para el Conocimiento y Uso de la Biodiversidad (REMIB, CONABIO)

…

DiGIR 4

Distributed Databases –Regional

Mountain and Plains Spatio-Temporal Database-Informatics (MaPSTeDI)

Ocean Biogeographic Information System (OBIS) Pacific Basin Information Node, National Biological

Information Infrastructure (PBIN, NBII) Species Link, Centro de Referência em Informação

Ambiental (Species Link, CRIA) A Virtual Herbarium of the Chicago Region (vPlants) Spatial Analysis of Local Vegetation Inventories Across

Scales (SALVIAS) …

DiGIR 5

Distributed Databases –Intra-institutional

Berkeley Natural History Museums (BNHM) Association of Biological Collections, UC Davis …

DiGIR 6

Distributed Databases –“Nodes”

LifeMapper National Biodiversity Information

Infrastructure (NBII) Global Biodiversity Information Facility

(GBIF)

DiGIR 7

GBIF Work Programmes NODES ECAT – Electronic Catalogue of Names of

Known Organisms DIGIT – Digitisation of Natural History

Collections OCB – Outreach and Capacity Building DADI – Data Access and Database

Interoperability

DiGIR 8

Taxonomic Database Working Group

Standards development and maintenance Access to Biological Collections Data (

ABCD) Darwin Core Version 2 (DwC2) Structure of Descriptive Data (SDD) DiGIR Others…

DiGIR 9

DiGIRDistributed Generic Information Retrieval

John Wieczorek, Stan Blum, Dave Vieglais, P.J. Schwartz

DiGIR 10

Information Retrieval Distributed - a protocol for retrieving

structured data from multiple, heterogeneous databases across the Internet.

Generic - a protocol independent of the data retrieved and of the software to retrieve it.

DiGIR 11

Project Rationale Avoid multiple incongruous development

efforts Pool resources and create a support

community of experts Solve scalability problems

DiGIR 12

Design Goals Use open protocols and standards, such as

HTTP and XML Decouple the protocol, software and

semantics Make new data provider installations as

easy as possible Develop open source software with GNU

General Public Licensing (It’s free).

DiGIR 13

DiGIR Component Summary

DiGIR 14

DiGIR ArchitectureProvider

DiGIR 15

Provider Receives requests Retrieves data from

database Sends results to requestor Supplies metadata to

describe content, contacts, and capabilities

Logs requests

DiGIR 16

DiGIR ArchitecturePortal Engine

DiGIR 17

Portal Engine The entry point for an

application Can query a registry to

discover potential providers Can determine, based on

provider metadata, whether a provider should be queried

Can send requests to multiple providers

DiGIR 18

Portal Engine, continued Assembles responses from

providers Returns packaged results to

the requesting application Communicates via protocol

compliant messaging only Logs activity

DiGIR 19

Registry Provides a “yellow pages” to

advertise the existence and capabilities of a provider

Provides a means to discover potential providers of interest

May be public or private Need not be a part of the

architecture

DiGIR 20

DiGIR ArchitectureProviderRegistry (register)

DiGIR 21

DiGIR ArchitecturePortal EngineRegistry (discover)

DiGIR 22

DiGIR Protocol Defines request and response message

formats for communication between provider, portal engine, and applications Metadata requests Search requests Inventory requests

Remains unfettered by the structure of the data it transfers

DiGIR 23

DiGIR ArchitectureApplication

DiGIR 24

DiGIR ArchitectureApplicationProtocol (request)Portal Engine

DiGIR 25

DiGIR ArchitectureApplicationProtocol (request)Portal EngineProtocol (request)Provider

DiGIR 26

DiGIR ArchitectureApplicationProtocol (request)Portal EngineProtocol (response)Provider

DiGIR 27

DiGIR ArchitectureApplicationProtocol (response)Portal Engine

DiGIR 28

Applications Must be able to assemble and send

a request document to a portal Must be able to receive and

interpret a response document from the portal

Must do something incredibly useful and interesting with the data

This is where the real fun is!

DiGIR 29

Hot topics – Interesting problems Persistent unique identifiers Web services (converters, translators, calculators,

transformation services) Data validation (taxonomic and geographic thesauri,

ecological niche modeling, expedition analysis, outlier detection,…)

Spatial query interfaces Data quality feedback mechanisms Automated georeferencing, event gazetteers Concept libraries, schema extensions, schema libraries,

federated ontologies

DiGIR 30

MaNIS – The Mammal Networked Information System

It’s more than just a pangolin…

DiGIR 31

MaNIS Network Configuration

LACMMS AccessDatabase

OnlineMS AccessDatabase

MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal

MVZ-MaNISPresentation

Layer

UMNH-MaNISPresentation

Layer

UWBM-MaNISPresentation

Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

CASSQL ServerDatabase

DiGIR 32




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 33




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 34




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 35




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 36




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 37




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 38




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 39




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 40




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 41




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 42




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 43




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database

UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider



DiGIR 44



MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database

UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider




DiGIR 45




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


UAMOracle

Database



DiGIR 46




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database

UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider



DiGIR 47




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 48




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 49




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database



Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


DiGIR 50




MVZSybase

Database

MaNISDiGIRPortal

LSUMZ4D-Mac

Database


UAMOracle

Database


MaNISDiGIRPortal

MaNISDiGIRPortal


Layer


Layer


Layer

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


…

…

…

…

… …

… …

DiGIR 51

“CalNet” Network Configuration



MVZSybase

Database

LSUMZ4D-Mac

Database


UAMOracle

Database


DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


CalNetDiGIRPortal

DiGIR 52

Event Gazetteer Network Configuration



MVZSybase

Database

LSUMZ4D-Mac

Database


UAMOracle

Database


BioGeomancerWeb

Service

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


CalNetDiGIRPortal

DiGIR 53

NBII Network Configuration



MVZSybase

Database

LSUMZ4D-Mac

Database


UAMOracle

Database


DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


NBIIDiGIRPortal

DiGIR 54

GBIF Network Configuration



MVZSybase

Database

LSUMZ4D-Mac

Database


UAMOracle

Database


GBIFPresentation

Layers

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider

DiGIRProvider


NBIIDiGIRPortal

DiGIR 55

Intra-Network Configuration (BNMH)

PHMAWorking

Database

OnlineDatabase

UCBGWorking

Database

DiGIRProvider

BNHMDiGIRPortal

UCJEPSWorking

Database

OnlineDatabase

UCMPWorking

Databases (4)

OnlineDatabase

EssigWorking

Database

OnlineDatabase

OnlineDatabase

BNHMPresentation

Layer

DiGIR 56

Other Network Configurations

WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRProvider

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

WorkingDatabase

OnlineDatabase

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRPortal

WorkingDatabase

DiGIR 57


WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRProvider

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

WorkingDatabase

OnlineDatabase

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRPortal

WorkingDatabase

DiGIR 58


WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRProvider

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

WorkingDatabase

OnlineDatabase

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRPortal

WorkingDatabase

DiGIR 59


WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRProvider

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

WorkingDatabase

OnlineDatabase

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRPortal

WorkingDatabase

DiGIR 60


WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRProvider

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

WorkingDatabase

OnlineDatabase

DiGIRPortal

WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRPortal

WorkingDatabase

DiGIR 61


WorkingDatabase

OnlineDatabase

DiGIRProvider

DiGIRPortal

WorkingDatabase

WorkingDatabase

DiGIR 62

Project Information DiGIR is a collaborative open source

development project on SourceForge (https://sourceforge.net/projects/digir).

Software and documentation are available on the DiGIR web site (http://digir.net).

MaNIS is an international network collaboration among mammal specimen collections (http://elib.cs.berkeley.edu/manis).

DiGIR 63

Hot topics – Interesting problems Persistent unique identifiers Web services (converters, translators, calculators,

transformation services) Data validation (taxonomic and geographic thesauri,

ecological niche modeling, expedition analysis, outlier detection,…)

Spatial query interfaces Data quality feedback mechanisms Automated georeferencing, event gazetteers Concept libraries, schema extensions, schema libraries,

federated ontologies

Distributed Databases and Applications Presentation

Documents

Transcript of Distributed Databases and Applications Presentation