Ariadne Harvesting
-
Upload
bram-vandeputte -
Category
Education
-
view
1.781 -
download
3
description
Transcript of Ariadne Harvesting
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
HARVESTING METADATA IN REPOSITORY
NETWORKS
Bram VandeputteK.U.LeuvenARIADNE
11
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Working since 2006 with Erik Duval
• ARIADNE harvester, OAI-PMH target support, validation service, etc.
• International projects : MACE, MELT, Globe, Ariadne, Aspect, iCoper,...
• Other interests : Information visualisation, Interoperability, TEL
2
WHO AM I
2
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• OAI-PMH
• Validation Service
• Harvester Infrastructure
• Projects
• MACE
• MELT
• GLOBE
• OAI-PMH Target3
Overview
3
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Server - Client model
• Pull mechanism
• options :
• selective harvesting
• incremental harvesting
4
OAI-PMH
4
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Verbs : Identify, ListRecords, GetRecord
• Parameters :
• baseUrl
• from & until date
• metadataPrefix
• sets
5
OAI-PMH
5
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Interoperability : Application Profile (AP)
• Manual check : very time consuming
• Need a tool for enforcing an AP => validation scheme
• A set of validation rules
• Reusable
6
Validation Service
6
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
7
Validation ServiceLOM loose
MELTv1.0/minimal
MELTv1.0/full
lomloose.xsd
conditional fields check
LRE taxon field numerical
vcard validator
empty attribute fields
MACEv1.0
classification vocab service
melt.xsd mace.xsd
Legend
uses
extends
MACEv1.0
vcard validator
validationScheme
validation component
7
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
88
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
The ARIADNE Harvester Infrastructure
88
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
harvester log
88
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
8
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
8
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
8
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
8
LOMLOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
8
LOM
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
Validation Msg
8
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
Validation Msg
8
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
Validation Msg
8
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
Validation Msg
8
LOM
SQISPI
8
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
OAI
External Repository
LOMLOM
OAI-PMH
LOM
LOMLOM
OAI-PMH
ARIADNE Harvester
validation serviceharvester log
ARIADNE Repository
Validation Msg
9
LOM
LOM
SQISPI
LOMLOM
OAI-PMH
LOM
Validation Msg
LOM
LOM
Validation Msg
1 2
34
5
6
9
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• MACE
• MELT
• GLOBE
10
Harvesting in networks
10
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Metadata for Architectural Contents in Europe
• www.mace-project.eu
• Harvesting, Searching, Enriching, (geo-) Tagging, attention metadata
11
MACE
11
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
Harvestedmetadata
WINDS
IRB
DYNAMO
ARIADNE Harvester
OAI-PMH
OAI-PMH
OAI-PMH
SPI
Enrichedmetadata
MACEmetadata
Uses Lucene for index search
CuminCAD
OAI-PMH
Validation service
SPI
SPI
12
MACE
12
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
13
MACE
13
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• MELT : a Metadata Ecology for Learning and Teaching
• Ministries of education across europe
• enriching, translating, better quality metadata
• practical test for scalability of harvester framework
14
MELT
14
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
15
MELT
MEC
MESC
MSU
NCTE
ARIADNE Harvester
Validation service
BMBWK
C-H
CFL
EDUCATIO
EUN
FWU
INDIRE
KUL
RVP
TLF
UL-FRI
XTECO
AI-
PM
H
SPI
ARIADNEMETADATASTORE
OAI
SQIARIADNE Harvester
(EUN)
EUN MDS
Enrichment Toolkit
Multilingual Thesaurus
Vocabulary Registry
EUN AMG
SQI
15
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
1616
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Global Learning Objects Brokered Exchange
• Worldwide network of networks of repositories
• => Both SQI and OAI-PMH are used on various levels in the network
17
GLOBE
17
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
18
GLOBEMEC
MESC
MSU
NCTE
ARIADNE Harvester
BMBWK
C-H
CFL
EDUCATIO
EUN
FWU
INDIRE
KUL
RVP
TLF
UL-FRI
XTEC
OA
I-P
MH
WINDS
IRB
DYNAMOARIADNE Harvester
CuminCAD
MELT
OAI
SQI
OA
I-P
MH
Federated
Search
EdNA CGIAR Pubelo Merlot
III COSL LORNET Ariadne
LACLO NIME EUN
ISKME KERIS
SQI UDDI
Registry
MACE
OAI
SQI
SQI SQI SQI SQI
SQI SQI SQI SQI
SQI Client
SQI
OAI
SQI
OAI
SQI
OAI
SQI
OAI
SQI
OAI
SQI
ARIADNE Harvester
GLOBE
OAI
SQI
SPI
18
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
1919
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
20
OAI-PMH target
20
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
21
OAI-PMH result
DYNAMO LOM metadataDYNAMO LOM
metadataDYNAMO LOM metadataRepository LOM
metadata
ARIADNEHarvester
1. Get metadata
out of database
Repository metadata
- identifier- title- url- project description- ...
Repository LOM metadata
- lom.general.identifier- lom.general.title- lom.general.description- lom.technical.location- ...
2. Mapping Process
4. Serve results
3. Copy results in
OAI-PMH service
OAI-PMH Target Software
Repository
21
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• Questions ?
22
どうも ありがとうございます!
22
ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb
• SPI : http://ariadne.cs.kuleuven.be/lomi/index.php/SimplePublishingInterface
• SQI : http://ariadne.cs.kuleuven.be/lomi/index.php/LorInteroperability
• IEEE LOM : http://ltsc.ieee.org/wg12/
• OAI-PMH : http://www.openarchives.org/
• OAI-PMH target wiki :http://ariadne.cs.kuleuven.be/lomi/index.php/Setting_Up_OAI-PMH
23
References
23