Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th...
-
Upload
nyah-galer -
Category
Documents
-
view
220 -
download
0
Transcript of Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th...
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
The SEALS Platform for Scholarly Research
Asun Gómez-Pé[email protected]
Ontology Engineering GroupUniversidad Politécnica de Madrid
Acknowledge to: Miguel Esteban Gutiérrez, Raul García-Castro, Nandana Mihindukulasooriya
and the SEALS consortium11.04.23
1
http://www.seals-project.eu/
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam2
Ed Hovy Requirements
• A kind of framework that graphically depicts the whole scholarly environment/workflow - on this, a set of markers, like Post Its notes, that each indicate a technology available for use at the relevant point - also on this, another set of Post Its notes that each indicate some kind of bottleneck or gap
• A kind of system that helps me to reproduce experiments that appear in research papers and compare my technology with other technologies
11.04.23
2
Asun Gómez-Pérez Requirements
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam3
11.04.23
3
Scenario
Tool1
Tool4
Tool5
Tool1
Tool4
Tool5
Tool1Tool2Tool1Tool2
Tool3Tool3 Tool1Tool4Tool5
Tool1Tool4Tool5
Tool1Tool1
Tool2Tool6Tool2Tool6
Tool6Tool6 Tool6Tool7Tool6Tool7
Scholarly workflows
How do I know which tools are appropriate?: Reproduce Experiments (unbiased, repeatable, …)
ToolsTools
Test dataTest dataEvaluationsEvaluations
ResultsResults
My toolMy tool My resultsMy results
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Heterogeneity in the tools to be compared
• Data Sets– Levels of complexity– Scalability– Synthetic, hand-crafted
and real world data• Tools
– Execution environment– Functionalities– Criteria to evaluate:
scalability, interoperability, etc.
• Metrics– Precision/recall, information
losses of data, etc.• Evaluations
– Fully automatic– The user is in the loop– Hybrid approach– Extrinsic and intrisic
• Starting point– From scratch– Extend and adapt
preliminar approaches
11.04.23
4
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Test dataselection
Metricsselection
Execution requirements
selection
Evaluation services
implementation
Interpretationselection
Goalsselection
Tool featuresselection
Toolsselection
Evaluation workflows
implementation
Assumptionsselection
Criteriaselection
Dimensions
Evaluationdesign
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
SEALS platform for technology evaluation
• SEALS Platform Features
6
• Open (everybody can use it)• Scalable (to users, data size)• Extensible (to more tests, different technology,
more measures)• Sustainable (beyond SEALS)• Independent (unbiased)• Repeatable (evaluations can be reproduced)
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam7
Test DataTest Data
EvaluationsEvaluations
ToolsTools
The SEALS platform
7
ResultsResults
11.04.23
Semantic Technology Recommendation
Framework
Results Comparisons
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam8
SEALS Service Manager
SEALS Service Manager
Runtime Evaluation
Service
Runtime Evaluation
Service
SEALS PortalSEALS Portal
Test Data Repository
Service
Test Data Repository
Service
Tools Repository
Service
Tools Repository
Service
Results Repository
Service
Results Repository
Service
Evaluation Descriptions
Repository Service
Evaluation Descriptions
Repository Service
Technology Providers
Evaluation Organisers
Technology Adopters
Software agents
SEALS Repositories
SEALS Logical Architecture
11.04.23
8
Virtualization:two execution environments
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Evaluation Execution Process at 10,000ft
9
ER
10.07.2012
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Evaluation services
10
Exploit resultsExploit results
11.04.23
ToolsTools
Test dataTest dataEvaluationsEvaluations
ResultsResults
Reproduce experiments
My toolMy tool My resultsMy results
My test dataMy test data My resultsMy resultsToolsToolsEvaluationsEvaluationsUse your dataset to evaluate others tool
ToolsTools Test dataTest data
My evaluationMy evaluation
My toolMy tool
My resultsMy results
My test dataMy test data
Define your own experiment
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam11
Achievements
11.04.23
Specification of the Platform Architecture
Workflows Implemented in BPEL
Test Data
Repository Service
Test Data
Repository Service
Tools Repository Service
Tools Repository Service
Results Repository Service
Results Repository Service
Evaluation Descriptions Repository
Service
Evaluation Descriptions Repository
Service
Implementation and Evaluation of Repositories
http://www.seals-project.eu/ontologies/
SEALS entities (dataset, tools, evaluations and results) are described using Ontologies
Semantic Technology Recommendation Framework
Results Comparisons
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam12
SEALS support to scholarly workflows
12
SEALS Service ManagerSEALS Service Manager
Runtime Evaluation Service
Runtime Evaluation Service
Test Data RepositoryTest Data Repository
Tools RepositoryTools Repository
Results RepositoryResults Repository
Evaluation Descriptions Repository
Evaluation Descriptions Repository
SEALS Repositories
Scholarly WorkflowRepository
Scholarly Portal
Tool1
Tool2Tool1
Tool2Tool3Tool3 Tool1
Tool4Tool5
Tool1
Tool4Tool5
Tool1Tool1
Tool2
Tool6Tool2
Tool6Tool6Tool6 Tool6
Tool7Tool6
Tool7Tool1
Tool4
Tool5
Tool1
Tool4
Tool5
Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam
Thank you for your attention!