Ontology Assessment – Proposed Framework and Methodology.
-
Upload
aiden-reynolds -
Category
Documents
-
view
214 -
download
1
Transcript of Ontology Assessment – Proposed Framework and Methodology.
Ontology Assessment – Ontology Assessment – Proposed Framework and Proposed Framework and
MethodologyMethodology
Which one is an ontology? Why/not? Which one is an ontology? Why/not?
Biological Classification Scheme
AMS Classification Scheme NASA Thesaurus Library of Congress Subject
Headings Dublin Core Metadata Scheme Organizational Chart ISO Country List Metadata repository scheme Master Data Repository Content architecture models
(OO models) SCORM
XML Schema for Directory Records Classification SchemeRecords Classification Scheme Social Network Social Network
RepresentationRepresentation FolksomonyFolksomony Domain Knowledge Map Domain Knowledge Map Visual representation of Visual representation of
concept clustersconcept clusters Financial ratios Economic indicators Mathematical formula XML structured electronic
journal issue WordNet
Why do we care whether one is or isn’t an ontology? What’s the point?
Personal RecommendationPersonal Recommendation
We need to distinguish between an ontology and applications that We need to distinguish between an ontology and applications that use ontologiesuse ontologies
We need to suspend our heavy reliance on different domain We need to suspend our heavy reliance on different domain terminologies that describe applications that use ontologies and terminologies that describe applications that use ontologies and adopt a neutral mental model adopt a neutral mental model
Minimize our references to dictionaries, synonym rings, Minimize our references to dictionaries, synonym rings, taxonomies, thesauri, class schemes, knowledge maps, content taxonomies, thesauri, class schemes, knowledge maps, content schema, etc. schema, etc.
Rather, compare the specific applications and standards to a neutral Rather, compare the specific applications and standards to a neutral framework – this will facilitate more intelligent conversations, and framework – this will facilitate more intelligent conversations, and will also help us to better communicate with others outside the fieldwill also help us to better communicate with others outside the field
Framework must reflect the multi-dimensionality of ontologies, Framework must reflect the multi-dimensionality of ontologies, though – a single, linear representation of applications does not though – a single, linear representation of applications does not serve as a framework serve as a framework
Goals of the FrameworkGoals of the Framework Before we can do this, though, we need to explicitly agree on the ‘end
game’ of a the framework
Do we agree that the goal is to develop a neutral, well defined, quantifiable, multidimensional framework against which any ‘thing’ that any one is calling an ontology could be evaluated?
Anyone who has anything they’re calling an ontology should be able to use the framework to judge:
whether it is or is not an ontology Which essential components it is missing Where it ranks on a scale of informal to formal ontologies What they can do to improve or enhance it
Enable assessment of any proposed ontology for the purpose of informing users about an ontology Providing developers with methodology for comparison and
improvement
Enable definition of: Minimum standards for what is/is not an ontology Thresholds for formal and informal ontologies
First StepsFirst Steps
If we agree on the goal, we need to start by defining the basic dimensions of an ontology.
Dimensionality proposed in the framework includes: Structure Expressiveness representational granularity intended use automated reasoning descriptive/prescriptive design methodology
Do the proposed framework dimensions accomplish this goal? Are these theoretical or practical dimensions? Do they work at a representational or on an analytical level? How easy would it be for people who are developing ontologies to understand them? Do they allow everyone who is working in an ontology space to play, or do they automatically
exclude some? Do they support the ‘end game’ of communication and use?
Framework RecommendationsFramework Recommendations
I would suggest that the framework still requires both simplification and I would suggest that the framework still requires both simplification and elaborationelaboration
Simplification in terms of how it groups factors, and elaboration in Simplification in terms of how it groups factors, and elaboration in terms of coverage of the factors that matter to those who are terms of coverage of the factors that matter to those who are developing or using ontologiesdeveloping or using ontologies
‘‘Pre-tested’ the framework with some colleagues in different areas of Pre-tested’ the framework with some colleagues in different areas of responsibility – none of them could understand the framework because responsibility – none of them could understand the framework because it was too theoreticalit was too theoretical
Need to bring it to a practical levelNeed to bring it to a practical level Need to describe the dimensions in terms of formal lists of factors and Need to describe the dimensions in terms of formal lists of factors and
concrete definitions for those factorsconcrete definitions for those factors
As we develop the framework, we must also define the analytical As we develop the framework, we must also define the analytical methodsmethods
The test of the framework is our ability to leverage it as an analysis method that The test of the framework is our ability to leverage it as an analysis method that allows us to neutrally characterize any ‘thing’ as an ontology and to be able to allows us to neutrally characterize any ‘thing’ as an ontology and to be able to explain the characterization so that anyone can understandexplain the characterization so that anyone can understand
Suggest that we should consider using a simple factor analysis for representation Suggest that we should consider using a simple factor analysis for representation and analysisand analysis
Simplify the FrameworkSimplify the Framework
I would suggest that the following framework accomplishes this ‘end game’ more effectively
Concepts – the nature of the content or values that are delivered or accessed through the ontology such as type, granularity, etc.
Relationships – nature, type, extent, specification of relationships, logic associated with relationships
Context – the context for which the ontology was developed and in which it may be used, including knowledge domain, application domain,
Governance – control and management of the concepts, relationships and context exercised by the developer or current user
Dimensions are orthogonal but yet sufficiently well defined that they allow Dimensions are orthogonal but yet sufficiently well defined that they allow us to include factors which are important to different kinds of ontology us to include factors which are important to different kinds of ontology applicationsapplications
Factor AnalysisFactor Analysis
Statistical method used to describe variability of factors in which the factors Statistical method used to describe variability of factors in which the factors are modeled as linear combinations. are modeled as linear combinations.
A single factor in the model would represent a set of ‘like’ variables which A single factor in the model would represent a set of ‘like’ variables which otherwise would be too complex to modelotherwise would be too complex to model
Factor analysis might help us to synthesize a set of variables into a single Factor analysis might help us to synthesize a set of variables into a single factor – to represent in this case a dimension of an ontologyfactor – to represent in this case a dimension of an ontology
Challenges: Challenges: Agree on dimensions (synthesis of factors)Agree on dimensions (synthesis of factors) Develop a method for quantifying factors appropriate to the dimensionDevelop a method for quantifying factors appropriate to the dimension Define the method of factor analysisDefine the method of factor analysis
Advantages: Advantages: it might help us to focus our discussions on actual factors and away from it might help us to focus our discussions on actual factors and away from
argumentationargumentation Allows everyone and anyone to play in the ontology spaceAllows everyone and anyone to play in the ontology space Allows everyone and anyone to characterize their ontology as a starting point for Allows everyone and anyone to characterize their ontology as a starting point for
conversations and interoperability conversations and interoperability We can keep the analysis simple since we are only using this to ‘characterize’ and We can keep the analysis simple since we are only using this to ‘characterize’ and
‘communicate’ – not to predict or to explain factors and ‘communicate’ – not to predict or to explain factors and
Example of Factor Analysis Example of Factor Analysis Methodology is currently used to calculate and visually display factors which
Contribute to the development or knowledge economies. Helps economists to compare and define knowledge economies.
Sample List of Innovation FactorsSample List of Innovation Factors
Let’s take as an example the ICT factor as it relates to Let’s take as an example the ICT factor as it relates to knowledge economiesknowledge economies
What factors might define the ICT Dimension?What factors might define the ICT Dimension? Access to computersAccess to computers Telecommunications developmentTelecommunications development Level of education achievedLevel of education achieved Investment in technology development (Tech R&D)Investment in technology development (Tech R&D)
Proposed Ontology Assessment Proposed Ontology Assessment MethodologyMethodology
Factor analysis for ontologies would involve …Factor analysis for ontologies would involve …
1.1. defining the essential dimensions of an ontologydefining the essential dimensions of an ontology2.2. defining those factors which characterize each dimension defining those factors which characterize each dimension 3.3. quantifying the factors quantifying the factors 4.4. analyzing the factors for any given application (factor analysis) or analyzing the factors for any given application (factor analysis) or
comparisoncomparison5.5. visually representing the analysis for a single ‘ontology’ and/or visually representing the analysis for a single ‘ontology’ and/or
for comparisons of ‘ontologies’for comparisons of ‘ontologies’
Let me explain how factor analysis might be usedLet me explain how factor analysis might be used If we can define the dimensions of an ontology, each dimension If we can define the dimensions of an ontology, each dimension
could then be represented as a composite measurecould then be represented as a composite measure The composite measure is made up of scores for a set of factors The composite measure is made up of scores for a set of factors
that define that dimension that define that dimension Having a composite score for each dimension would allow us to Having a composite score for each dimension would allow us to
use a very simple analytical method that would characterize or use a very simple analytical method that would characterize or compare specific ontology applicationscompare specific ontology applications
Representation of Ontological Representation of Ontological AssessmentsAssessments
Methodology could be used to generate a factor index for ontologies, to rank and compare ontologies.
Dimension 3 Dimension 2
Dimension 1
Dimension 4
Another Dimensionality Framework
Index of Factors
Index of Factors
Index of Factors
Index of Factors
Factor AnalysisFactor Analysis
Factor analysis could be conducted:Factor analysis could be conducted: At the component level on that subset of factorsAt the component level on that subset of factors At the ontology level, across all factorsAt the ontology level, across all factors
Developers or users could determine what the optimal Developers or users could determine what the optimal dimensionality was for their particular use dimensionality was for their particular use
Summit members and the Ontology community could Summit members and the Ontology community could identify minimun factor scores that define what is/is not an identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontologyontology, and what constitutes a full, formal ontology
Ultimately, this may provide us with an ecumenical vs. Ultimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development evangelical approach to ontological standards development and assessment and assessment
Representation of Ontological Representation of Ontological AssessmentsAssessments
Structure
RepresetationalGranularity
Expressiveness
Descriptive vs.Prescriptive
Dimensionality Suggested in the Framework Paper
Intended Use
Use of AutomatedReasoning
Critical Question: Are these dimensions orthogonal, mutually exclusive and clean enough for analysis?
Representation of Ontological Representation of Ontological AssessmentsAssessments
Methodology could be used to generate an ontological factor index for ontologies, and to rank and compare ontologies.
Concepts Context
Relationships
Governance
Another Dimensionality Framework
Representation of Ontological Representation of Ontological AssessmentsAssessments
Methodology could be used to generate an ontological factor index for ‘ontological things’, and to rank and compare ontologies.
Concepts Context
Relationships
Governance
Sample assessment of a folksonomy
Representation of Ontological Representation of Ontological AssessmentsAssessments
Concepts Context
Relationships
Governance
Sample assessment of a medical disease classification scheme
Representation of Ontological Representation of Ontological AssessmentsAssessments
Concepts Context
Relationships
Governance
Sample assessment of an institutional records classification scheme
Defining and Quantifying FactorsDefining and Quantifying Factors For each component an orthogonal, independent set of For each component an orthogonal, independent set of
factors must be defined factors must be defined
Factors must be independent of any particular pre-existing Factors must be independent of any particular pre-existing ontology (neutral)ontology (neutral)
Each factor must have a quantifiable method of Each factor must have a quantifiable method of representation that lends itself to ‘scoring’, analysis and representation that lends itself to ‘scoring’, analysis and comparisoncomparison
Factors must have agreed upon definitions, be easily Factors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in interpreted by people and machines, and be inclusive in their coverage of values/conditionstheir coverage of values/conditions
To illustrate the idea, selected examples are presented in To illustrate the idea, selected examples are presented in following slidesfollowing slides
Selected Examples of Concept FactorsSelected Examples of Concept Factors
Concept typesConcept types Data/numbersData/numbers Calculation/ratiosCalculation/ratios WordsWords Grammatical fragmentGrammatical fragment Logical statementLogical statement Rule expressionRule expression Engineering equationsEngineering equations
Degree of ambiguityDegree of ambiguity Context Context
sensitivity/insensitivity sensitivity/insensitivity of definitionof definition
Representational formRepresentational form Usable encoding Usable encoding
methodmethod Availability of Availability of
representational representational specifications (Strings specifications (Strings vs. syntax)vs. syntax)
Degree of Degree of conceptualization/ conceptualization/ specificationspecification Theoretical to commitalTheoretical to commital
What else…?What else…?
Selected Examples of Relationship Selected Examples of Relationship FactorsFactors
Simple expressive form Simple expressive form of relationshipsof relationships GrammaticalGrammatical MathematicalMathematical LogicalLogical
Relationship behavior Relationship behavior Membership dependenceMembership dependence Representation or Representation or
instanceinstance EquivalenceEquivalence Causal dependenceCausal dependence Derivational dependenceDerivational dependence
Degree of Relationship Degree of Relationship Validation/RigorValidation/Rigor Fully SubjectiveFully Subjective Grammatical validationGrammatical validation Mathematical validationMathematical validation Logical rigor/validationLogical rigor/validation
What else?What else?
Selected Examples of Context FactorsSelected Examples of Context Factors
Knowledge ContextKnowledge Context Formal vs. informal Formal vs. informal
knowledge domainknowledge domain
Application ContextApplication Context System vs. human System vs. human
application/ application/ consumptionconsumption
Managed/standardized Managed/standardized application vs. home application vs. home growngrown
Functional contextFunctional context SearchSearch Mathematical or Mathematical or
statistical analysisstatistical analysis Logical inferenceLogical inference ClassificationClassification Dynamic clusteringDynamic clustering Metadata representationMetadata representation Concept indexingConcept indexing
What else…?What else…?
Selected Examples of Governance Selected Examples of Governance FactorsFactors
Standards AvailabilityStandards Availability Published formal vs. Published formal vs.
guidelines vs. ad hoc guidelines vs. ad hoc conceptsconcepts
Published formal vs. Published formal vs. guidelines vs. ad hoc guidelines vs. ad hoc relationshipsrelationships
Prescriptive vs. Descriptive Prescriptive vs. Descriptive GovernanceGovernance Enforcement of standardsEnforcement of standards
Design GuidelinesDesign Guidelines Top-down (model) vs. Top-down (model) vs.
Bottom-up (empirical)Bottom-up (empirical)
ExtensibilityExtensibility Degree to which others Degree to which others
can add to or extend can add to or extend either the concepts or the either the concepts or the relationshipsrelationships
CurrencyCurrency Degree to which the Degree to which the
concepts and/or concepts and/or relationships represent our relationships represent our current view or knowledge current view or knowledge of the contextof the context
What else…?What else…?