The Evaluation of Ontologies Editorial Review vs. Democratic Ranking
description
Transcript of The Evaluation of Ontologies Editorial Review vs. Democratic Ranking
The Evaluation of Ontologies
Editorial Review vs. Democratic Ranking
Barry Smith
1
The need for ontology evaluation
Ontologies are expensive.
$8 million have been invested in the Gene Ontology thus far
Has this investment been worthwhile?
Are some ontologies more useful than others?
What are ontologies useful for ?2
Uses of ‘ontology’ in PubMed abstracts
3
In the olden days
people measured lengths using inches, ulnas, perches, king’s feet, Swiss feet, kanejaku, shaku, whale shaku, etc., etc.
5
then, on June 22, 1799,everything changed
we now have the International System of Units
Through the SI System
science becomes a cumulative, distributed endeavormy measuring equipment can be callibrated against your measuring equipmentmy hypotheses can be checked against your data
When should a new unit should be included in the SI system?
The work of the
CIPM International Committee for Weights and Measures
rests on an editorial process
9
Obvious benefits of a peer review process for scientific work
Creating an environment which rewards better work
Helping people to find better work
...
10
Proposal: Evaluate ontologies via an editorial process
of the sort used for scientific journals and scientific research projects
a process of peer review by human experts
11
Peer review process
• appropriate when evaluating science
• (not, e.g., when evaluating poetry, or fairy tales, or Chinese mythology ...)
12
Peer review process
appropriate in the domain of biomedical ontology
– ontology evaluation here can be of particularly acute concern
13
14
RELATION TO TIME
GRANULARITY
CONTINUANT OCCURRENT
INDEPENDENT DEPENDENT
ORGAN ANDORGANISM
Organism(NCBI
Taxonomy)
Anatomical Entity(FMA, CARO)
OrganFunction
(FMP, CPRO) Phenotypic
Quality(PaTO)
Biological Process
(GO)CELL AND CELLULAR
COMPONENT
Cell(CL)
Cellular Compone
nt(FMA, GO)
Cellular Function
(GO)
MOLECULEMolecule
(ChEBI, SO,RnaO, PrO)
Molecular Function(GO)
Molecular Process
(GO)
The OBO Foundry
THE OBO FOUNDRY
15
a suite of reference ontologies in the biomedical domain satisfying certain basic criteria and subject to an on-going process of peer review
OBO FOUNDRY CRITERIA
16
The ontology is open and available to be used by all.
The ontology is in, or can be instantiated in, a common formal language.
The developers of the ontology agree in advance to collaborate with developers of other OBO Foundry ontology where domains overlap.
17
CRITERIA UPDATE: The developers of each ontology
commit to its maintenance in light of scientific advance, and to soliciting community feedback for its improvement.
ORTHOGONALITY: They commit to working with other Foundry members to ensure that, for any particular domain, there is community convergence on a single controlled vocabulary.
18
CRITERIA
IDENTIFIERS: The ontology possesses a unique identifier space within OBO.
VERSIONING: The ontology provider has procedures for identifying distinct successive versions.
The ontology includes textual definitions for all terms.
CRITERIA
19
CLEARLY BOUNDED: The ontology has a clearly specified and clearly delineated content.
DOCUMENTATION: The ontology is well-documented.
USERS: The ontology has a plurality of independent users.
CRITERIA
20
COMMON ARCHITECTURE: The ontology uses relations which are unambiguously defined following the pattern of definitions laid down in the OBO Relation Ontology
CRITERIA
The OBO Foundry provides guidelines (traffic laws) to new groups of ontology developers in ways which can
• help to ensure interoperability through prospective synchronization
• counteract dispersion of effort • prevent some common types of
nonsense21
22
Example: the Foundry seeks orthogonality
This brings division of labor and other benefits
Foundry editors adjudicate in areas of overlap
How the editorial process works
23
Foundry editors balance– the flexibility that is indispensable to
scientific advance
– the institution of principles that is indispensable to successful coordination
How the editorial process works
24
is a top down approach, relying on authority
Peer review
25
democratic ranking
An alternative, bottom up approach
26
If we build it, will they come? Social engineering of new technology to
disseminate biomedical ontologies
Mark A. Musen and the BioPortal Team
Stanford University
27
If we build it, will they come? Social engineering of new technology to
disseminate biomedical ontologies
presentation to Ontolog Forum
July 6, 2007
28
With thanks to
Mark Musen and Natasha Noy
29
30
In biology, lots of ontology developers are almost hobbyists
• Nearly always, ontologies are created to address pressing practical needs
• Biologists ... may have little appreciation for metaphysics, principles of knowledge representation, or computational logic
• There simply aren’t enough good ontologists to go around
31
Issues in assuring ontology quality• Unlike the case with journal submissions, it makes no
sense for ontologies to be peer-reviewed by just a handful of experts
• Open, community-based review of ontologies may be haphazard and chaotic
• Top–down solutions may offer rigid review critieria at the expense of scalability
• There is a pressing need for empirical evaluation of methods for ontology evaluation
32
OBO Foundry must address lots of questions
• Can the top-down approach scale? How many ontologies can be managed by a small panel of curators?
• Who gets to reject an ontology on the basis of form or content? What is the appeals process? How do we know whom to believe?
• Who will curate the curators?
33
NCBO will offer
• Technology for uploading, browsing, and using biomedical ontologies
• Methods to make the online “publication” of ontologies more like that of journal articles
• Tools to enable the biomedical community to put ontologies to work on a daily basis
34
http://bioportal.bioontology.org
35Local Neighborhood view
Browsing/Visualizing Ontologies
36
Hierarchy-to-root view
38
Goals for BioPortal
• Web accessible repository of ontologies for the biomedical community– Archived locally– Anywhere in cyberspace
• Support for ontology– Peer review– Annotation (marginalia)– Versioning– Alignment– Search
39
Ontologies are not like journal articles
• It is difficult to judge methodological soundness simply by inspection
• We may wish to use an ontology even though some portions – Are not well designed– Make distinctions that are different from
those that we might want
40
Ontologies are not like journal articles
• The utility of ontologies– Depends on the task– May be highly subjective
• The expertise and biases of reviewers may vary widely with respect to different portions of an ontology
• Users should want the opinions of more than 2–3 hand-selected reviewers
• Peer review needs to scale to the entire user community
41
Community-Based Annotation
• Makes ontology evaluation a democratic process
• Assumes users’ applications of ontologies will lead to insights not achievable by inspection alone
• Assumes end-users will be motivated to comment on and engage in dialog about ontologies in the repository
Solution Snapshot
45
Open ratings for ontologies• Any user can
– rate an ontology– add a “marginal note”
• Ontology evaluation becomes a community-based initiative
• A web of trust can enable users to filter comments or ratings to avoid “noise”
Arguments in favor of the top-down approach in the scientific domain
• marginalia will contain a great deal of irrelevantalia
• scientists need ontologies, but are normally not experts in ontology; they are looking for authoritative guidelines
• the Foundry process is yielding guidelines on how to build ontologies compatible with those which already exist
46
Arguments against the top-down approach
• ontologies are not like journal articles, and it is difficult to judge methodological soundness simply by inspection.
• the evaluation process does not yield a quantifiable result.
– but scientific journals face exactly similar problems, yet peer review, there, works well
47
In defense of democratic rankings
• ranking by large numbers of users will tend to counteract such biases (but will the ranking service in fact attract users?)
• ranking by large numbers of users has a greater opportunity to scale up when ontologies proliferate
48
We have common goals
• Both approaches seek quality assurance to support ontology selection.
• Both approaches need to address the fact that the expertise and biases of reviewers may vary widely with respect to different ontologies or to different portions of an ontology.
49
One big difference
For Musen et al. there are no restrictions on entry
The bottom-up approach seeks community-based annotation of ontologies, with no difference being made between experts and non-experts
50
One big difference
In the OBO Foundry reviews are created precisely by the peers of the ontology authors themselves—by persons with established and recognized expertise and with a demonstrated willingness to invest due diligence in ontology development, use, and evaluation.
51
Both are needed
• in domains such as refrigerators
• but in science?
52