Formative Evaluation of Ontologies for Information Agents

8/4/2019 Formative Evaluation of Ontologies for Information Agents

1/6

FORMATIVE EVALUATION OF ONTOLOGIES FOR INFORMATION

AGENTS

Muthukkaruppan Annamalai

Faculty of Computer Science and Software Engineering, MARA University ofTechnology, 40450 Shah Alam, Selangor, Malaysia ([email protected])

ABSTRACT

Information agents require knowledge about a particular

domain to gather useful information for humans. This

knowledge should ideally be stored externally in

ontologies. An application developer either builds or

simply adopts and extends existing domain ontologies to

describe the applications resources in meaningful ways

to information agents. In order to realise this potential,

the domain knowledge should be explicitly representedwithin `good ontologies. The question is how do we

know if an ontology being evolved is good, i.e., how do we

know if an ontology is appropriate for particular need? If

not, what is lacking in it and how it can be enhanced to

make it suitable for use? The answer emerging in the

knowledge engineering literature is ontology evaluation.

The evaluation of an ontology is crucial for its successful

use and reuse, yet it is acknowledgingly a difficult issue to

tackle. Up to now few works have been performed to this

end. This paper addresses the issue of formative

evaluation (validation and verification) of the ontologiesbeing developed for information agents. In particular, it

critically reviews the existing criteria based evaluation

practices to advance a more feasible set of criteria,

making additional distinctions about the aspects of the

ontology that came under the purview of each of the

evaluation criterion.

1. INTRODUCTION

Information agents can search and gather specific pieces

of information from the distributed knowledge sources in

a networked environment. The ability of an information

agent to perform this task effectively rests with the

knowledge it holds about the domain. This adaptable

knowledge must be described in a way that the

information agents can use during search and retrieval,

and hence warrants an ontology.

An ontology is a representation vocabulary,

specialised in some domain or subject matter. The

ontology defines a set of concepts for representing

specific facts in an instance of a domain (Chandrasekaran

and Josephson 1999). The emphasis is on the

conceptualisation of the conceptual terms in the

vocabulary intended to be captured. Consequently, the

notion that ontology is an explicit specification ofconceptualisation in a shared domain as proposed by

Gruber (1993) is the widely cited definition in artificial

intelligence (AI) and information systems (IS). Ontology

serves as a primary source for structuring and annotating

knowledge content, and subsequently provides machine-

readable semantic knowledge about the information in the

knowledge sources to the agent. A shared ontology

additionally facilitates a knowledge community to

exchange information in a domain.

A good domain ontology represents the modelled

world closely through consistently and coherentlyspecified definitions of concepts and relationships that

hold among them. It directs an information agent to infer

precise information from the knowledge contents that are

conceptually linked to this ontology. In contrast, an

ontology containing erroneous and incomplete definitions

does not provide a reliable semantic basis for applications

that depend on it. Consequently, a formative evaluation

of an ontology can help to identify its inherent

deficiencies that can be addressed while the ontology is

being developed to produce an effective conceptualisation

of the knowledge in the domain of discourse.

Unfortunately, the research and development onshared domain ontologies for information agents is still

largely confined to the academic environment. Since

researchers are more concerned about the technology that

utilises an ontology, passable ontologies are seen as

sufficiently meeting their current research needs. As

such, evaluating the goodness of domain ontologies has

not been a primary interest of researchers. This is evident

from the very small number of papers that have been

written about ontology evaluation so far. However, we

believe this lackadaisical attitude is set to change with the

advent of agents to facilitate knowledge enhanced search

and information retrieval. This will spur the production

of good domain ontologies for a wide range of application

domain. Formative evaluation plays a key function in the

development of a good ontology. The evaluation

corroborates the reliability of the ontology being built,

i.e., whether the conceptual definitions in the ontology are

well-founded. If not, the evaluation effort prompts us to

consider what is needed to produce an effective

conceptualisation that could adequately represent the

knowledge in the domain of discourse.

A domain ontology is often designed with the

potential of reuse in various applications in a domain area.

Therefore, a seemingly time saving strategy to create a

new ontology for describing a specific situation in adomain is by simply adopting an existing one. However,


2/6

in reality the domain specific knowledge needed by an

agent varies from application to application. Some

applications require more or less detailed knowledge

about a domain than others. As such, the evaluation of

the selected ontology is a precursory step to decide

whether that ontology is suitable for adoption. The

formative evaluation will expose the ontology's strengthsand weaknesses, and suggests the modifications and

extensions that are necessary to make the ontology

amenable to use.

In principle, an ontology should be evaluated on the

syntax, structure and semantics of its conceptual

definitions. The syntactic correctness of the definitions

could be checked using syntax analysers incorporated

within editors, parsers and validators. There has been

considerable work developing such tools. Discussing

them is beyond the scope of this paper, but information

about some of these tools can be found at ontology

language websites such as http://www.daml.org/tools/tools.html/ and http://www-ksl-svc.stanford.edu/ .

During the development of an ontology, we are mainly

concerned with the evaluation of the structure and

semantics of the conceptualised entities. In this paper, we

present a set of criteria to assess the structural and

semantic adequacy of a domain ontology in the context of

its applications in a domain area. In Section 2, we give a

brief description about ontologies for information agents.

In Section 3, we critically review the existing criteria

based evaluation practices to propose a more feasible set

of ontology evaluation criteria. The evaluation of

ontologies with respect to the prescribed set of evaluation

criteria is reviewed in Section 4. Finally, the concluding

section we summarise the contribution of this paper and

point to the direction of future work.

2. ONTOLOGIES FOR INFORMATION AGENTS

An ontology for information agents is a library of related

concepts, explicitly defined and formally organised into

subclasses, as a way of structuring and defining the

meaning of the represented conceptual terms. In general,

the ontology specifies classes of entities (concepts) in the

domain, relationships between these concepts and

properties attributed to them. Rigorously structuredontologies also describe functions and axioms (rules)

associated with these concepts to further constrain their

interpretation. In the simplest case, however, an ontology

is structured as a hierarchy of concepts related by

subsumption (subtype-supertype) relationships, reflecting

a taxonomy of conceptualisation.

3. RELATED WORK AND DISCUSSION

The importance of ontology evaluation is recognised in

existing literature on ontology development. At the same

time, it is commonly acknowledged that evaluating thequality of an ontology is a difficult issue to tackle.

Gruninger and Fox (1995) and, Uschold and King

(1995) highlight the evaluation of the reliability of a

developed ontology. The former uses a set of competency

questions to evaluate the suitability of an ontology, while

in the latter, the ontology is evaluated against its

requirement specification.

The importance of ontology evaluation is alsostressed by Fernandez et. al. (1997), except that the

evaluation is to be carried out as part of the ontology

development process; this paper shares a similar view.

A more serious effort to evaluate ontologies was

initiated by Gomez-Perez (1996). Drawing inspiration

from evaluation of knowledge systems, she emphasises

both validation and verification of the ontology content

are two important aspects of ontology evaluation. In

retrospect, the origin of these terms can be traced to

evaluation of software (Boehm 1981), the precursor for

knowledge systems. Validation and verification are

associated with checking the appropriateness and thecorrectness of the developed software, respectively.

Adapted to the evaluation of ontologies, validation checks

the suitability or appropriateness of the ontology being

designed, and verification checks the correctness and the

adequacy of the ontology.

The ontology is evaluated against its frame of

reference to ensure that it appropriately and adequately

satisfies its purpose of design. Without such frames of

references, it is difficult to check if the ontology is in

compliance with its purpose of design, i.e., conforms to

the understanding of what it is to be used for, and

competently satisfies its needs and requirements. The

common frames of references are requirement

specification, competency questions and, knowledge and

information resources in the subject domain.

Subsequently, the researchers in ontology

development have recommended numerous evaluation

criteria, on which the technical evaluation of the

ontologies can be based in order to guide the development

of ontologies having requisite quality of design. In

essence, the evaluation is a subjective means to check the

compliance of the ontology with respect to certain desired

properties in ontologies.

From the perspective of modelling in AI, Gruber

(1995) advocated the use of criteria such as clarity,coherence, extensibility, minimal encoding bias, and

minimal ontological commitment, which serve as

precedent for the design of an ontology. These notions

reverberate in contemporary ontology development

methodologies. For example, Uschold (1996) prescribed

that an ontology is evaluated using criteria as clarity,

consistency and reusability, which in retrospect, a set of

design criteria. In the same vein, Fox and Gruninger

(1998) suggested the use of criteria as perspicuity,

precision, generality, granularity, minimality

(conciseness), expressiveness and competence. Gomez-

Perez (1999, 2001) proposed consistency, completeness,conciseness and expansibility as worthy evaluation

criteria.


3/6

The modelling of knowledge in AI is strongly similar

to the modelling of the entities and relationships in the

conceptual schema. From the perspective of IS, the

entity-relationship model alludes to design qualities as

correctness, completeness, understandability, simplicity,

integration and implementability (Moody and Shanks

1994). The aforementioned design qualities cover muchof the same kind of ground as the evaluation criteria

mentioned in the previous paragraph.

It is obvious that the above researchers tend to

distinguish different set of criteria based on different

concerns in the design and evaluation of an ontology. In

spite of that, the quality demanded of some criteria

overlap with many others. We cite the following as

examples: Coherence and consistency correspond with

correctness; Clarity extends along with understandability,

perspicuity and expressiveness; Granularity and precision

accede to competency; Minimal ontological commitment,

generality and reusability are conformable criteria.However, accepting all of the above as valid design

criteria will give rise to conflict. For example, the

admittance of the level of expressiveness offered by

rigorous formalism for proper interpretation of defined

concepts, clashes with a simultaneous call for simplicity.

Likewise, there is a friction between clarity and

conciseness. The discord between reusability of the

modelled knowledge and its usability is another matter to

take issue.

We deem it is essential to bring together a

consolidated set of evaluation criteria that are compatible

with each other. Of the utmost necessity is the conceptual

definitions must be semantically consistent and coherent

with the modelled area of knowledge, and the definitions

must be consistent with each other, i.e., none of the

conceptual definitions in the ontology contradicts with the

other. Since, global consistency is not easy to enforce, we

should at least strive to achieve local consistency, i.e., an

ontology that is consistent with the view of a particular

user-community. An effective ontology must be

adequately complete to satisfy its needs and requirements.

An ontology void of unnecessary details is free from

ambiguity and redundancy, and additionally facilitates its

checking and maintenance. Because ontology

development is an evolutionary process, the design mustallow for incremental modifiability, i.e., it must be able to

be monotonically extended, conserving the existing

conceptualisation. Therefore, we regard consistency and

coherence, completeness, conciseness and, extensibility

and expansibility as important evaluation criteria of

ontologies.

A useful ontology should be sufficiently expressive,

whose concepts are defined at the appropriate level of

details and is adequately competent in meeting its purpose

of design. Representing the conceptualised model with

minimal encoding bias is yet another practical design

criteria. Hence, we set down expressibility, minimalencoding bias and competency as additional evaluation

criteria, which brings about the following discussion on

the impending design trade-offs.

On the trade-off between simplicity and

expressibility, we concur with the general view that

ontologies should be kept simple so that they are easy to

implement that encourages its reuse (Staab 2002).

Nevertheless, we will still argue that an ontology must besufficiently expressive to appropriately constrain the

possible interpretation of the conceptual terms that meets

its original purpose of design.

We also concur with Gruber that an ontology should

as far as possible be represented independent of symbol-

level encoding. Representing an ontology at an

intermediate level apart from its impending

implementation allows for the specification of

conceptualisation that is not severely restricted by

particular language limitation.

In relation to competency, there is a trade-off

between reusability and usability of an ontology. Themodelling for usability is directed by a purposive

mechanism, so that a defined concept can be applied more

directly to specify related knowledge content. On the

other hand, the modelling for reusability emphasises

generically represented vocabulary that is applicable

across many domains in a variety of situations.

Ontologies developed following the latter fashion of

design assert minimal ontological commitments, thus

calling for another design decision to make the necessary

extension and rework before they can be usefully applied.

Further more, the generic terms in the extended ontology

tend to cloud the conceptualisation with extraneous

knowledge that may be of little or no interest to the

knowledge community. Therefore, we argue that working

towards adequate ontological commitment to support the

desired competency should be viewed as an integral

aspect of ontology design, such that the ontology is at

disposal for use.

Hence, we propose consistency and coherence,

completeness, conciseness, extensibility and expansibility,

competency, expressiveness and minimal encoding bias as

a credible set of ontology evaluation criteria for

information agents.

4. ONTOLOGY EVALUATION CRITERIA

In this section, we discuss the evaluation of ontologies

with respect to the prescribed set of ontology evaluation

criteria; and making additional distinctions about the

aspects of the ontology that came under the purview of

each of the evaluation criterion.

4.1. Consistency and Coherence

The most important criteria in ontology evaluation are to

verify the consistency and coherence of the conceptual

definitions in the ontology. The verification checks onthe correctness and relevance of the concepts and the

properties associated with them. It involves two levels of


4/6

analyses. Firstly, we analyse the structural integrity of

individual conceptual definitions to check if each concrete

concept in the ontology corresponds to a specific entity in

the modelled area of domain knowledge, and the

definition is logically consistent with the intended

conceptualisation. We also check to ensure that each

individual assertion (instantiated concept) in the ontologyis true. Secondly, we analyse the definitions in the

ontology as a whole, which includes the abstract concepts.

Collectively, the definitions in the ontology must also be

consistent and coherent with each other, such that the

definitions in the ontology do not contradict with one

another.

4.1.1. Individual Conceptual Integrity

An ontology is comprised of a restricted set of concepts

related to an area of knowledge. For each concept, a set

of properties, i.e., attributes and relations linking it toother concepts are defined to facilitate its interpretation.

Information is used to crystallise a concrete concept's

structure with the appropriate semantic content such that

an instantiated concept can carry specific information

about the entity it represents.

The conceptual integrity verifies the individual

conceptual definitions with respect to the information

used to articulate a concept's structure. We also check to

ensure that there is no contradiction in the interpretation

of a concrete concept with respect to the entity it

represents.

4.1.2 Collective Consistency

Collective consistency analyses the coherence of the

definitions in the ontology by verifying the relationships

that bind the concepts. We verify the specialisation,

generalisation and equivalence relationships depicted in

subsumption hierarchical classification, the commonly

used scheme for structuring concepts in ontologies. We

also review the association or cross relationships that exist

between the concepts in the ontology, as well as the

reciprocal relationships that complement the cross

relationships such as inverse and transitive relationships.

The idea is to check whether the logical relationshipsdefined in the ontology intuitively reflect the

dependencies between their corresponding entities (both

concrete and abstract) in the modelled area of knowledge.

Like the concept structures, the relations can also be

structured in the ontology. A relation can be defined as a

specialised or generalised form of another relation; or it

can be specified as equivalent-to or inverse-of another

relation. In such case, the relational dependencies ought

to be analysed separately.

4.2. Completeness

If consistency and coherence criteria are used to verify the

correctness and relevancy of the concepts and the

properties defined in the ontology, the completeness

criterion checks whether the ontology has covered all the

relevant concepts and properties. Gomez-Perez (1999)

goes a step further by stating that an ontology is deemed

complete if each definition in the ontology is complete

with respect to the real world. This additional requirement

is voiced in a philosophical tone and demands that eachdefinition in the ontology is analysed with respect to its

coverage of the knowledge in the real world -- a

requirement that is hard to accomplish. Ontologies are by

nature incomplete; and we uphold the view that a domain

ontology is not a representational mechanism to answer

arbitrary questions about the domain.

Based on the above argument, our completeness

evaluation will go as far as to assess the functional

adequacy of the ontology in the context of its use and the

purpose of its design. The functional needs are stipulated

in the requirement specification and described by a pre-

generated set of competency questions. So, when we sayan ontology is complete, we actually mean the ontology is

functionally complete with respect to the area of

knowledge implied by its reference framework, rather

than ontologically complete with respect to the real world.

We will see later that a functionally complete ontology is

the basis for evaluating the competence of an ontology.

A functionally complete ontology meets all the

requirement specification of the modelling needs and/ or

is able to characterise the answers of all the competency

questions using the terms defined in the ontology. The

competency questions play an eminent role in the

completeness analysis of the modelled purposive domain

knowledge. The ontology needs to support not only the

statement of the answer, but also the derivation of the

answer for each of the competency questions. The

incompleteness of an ontology becomes evident when

either the requirement specifications are not satisfied or

the ontology cannot express the answer(s) to a

competency question posed to it. To address the

deficiency in the ontology, the missing concepts and the

properties have to be accounted for.

4.3. Conciseness

In contrast to the verification of completeness, theverification of conciseness checks to see whether there are

extraneous definitions in the ontology. This analysis

seeks to identify and remove redundant definitions that

are present in the ontology. In general, unnecessary and

unwanted definitions do not add value to the ontology.

Obvious redundancies are redefinition of existing

concepts in an ontology. Particularly, redefinition of

equivalent concepts using separate set of properties must

not be allowed because they are susceptible to differing

interpretation and use. The redefinitions that introduce

polysemous concepts can seriously impair the reliability

of the ontology. We believe such redundancy could beavoided by questioning the justification behind the

inclusion of each definition in the ontology. A more


5/6

demanding task is to check for inferred redundancies in

the ontology, i.e., whether a result can be deduced in more

than one way.

The point is to remove concepts and properties whose

presence in the ontology cannot be justified.

4.4. Extensibility and Expansibility

While an ontology carving a specific area of knowledge

can be shaped as a separate entity, it must be easy to

extend the ontology and expand its conceptual definitions.

The idea is to allow for incremental modification,

specialisation and adaptation of the ontology without

having to revise the existing definitions.

Extensibility checks to ensure that the concepts are

defined and organised in the manner that facilitate future

extension of the ontology. Expansibility checks if the

ontology allows expansionary modification such as

addition of new conceptual definitions and expansion ofexisting definitions without degenerating its present state

of being.

Clearly, the formalism in which an ontology is

represented is a tangible factor determining its

extensibility and expansibility. For example, the

hierarchically organised, objectified concept definitions

constructed using Frames provides a framework for

ordered representation that are easy to read and change,

and are also able to scale more easily as compared to say,

logical rules (Grosso et. al. 1999).

4.5. Competency

The competency criterion is specifically used to assess

whether the concepts in the ontology are defined with the

requisite level of detailness. The check on competency

helps to ensure that an ontology is capable of supporting

the purpose of its design. In general, the concepts in such

ontologies ought to be represented at a finer level of

granularity. The level of details that must be captured by

a conceptual definition is however dependent upon the

needs and requirements that arise in order to competently

utilise the modelled knowledge.

As a general rule, a competent ontology must also be

functionally complete. The detail representation offersdirect assistance for immediate adoption of the ontology.

So, in addition to the verification of completeness with

respect to the ontology's frame of reference, we also need

to verify the competency of an ontology by checking on

its ability to express relevant content of the knowledge

artefacts.

4.6. Expressiveness and Minimal Encoding Bias

As point out in Section 2, an ontology for information

agents must be specified with a certain degree of

formality. The ontology must be expressed using aformalism that is easy to read and understand so as to

facilitate its evaluation, maintenance, implementation and

uptake. In other words, the ontologies must be accessible

to both human and formal tools.

Although expressiveness and minimal encoding bias

are design decisions that is often factored into the

ontology representation formalism, an intermediate, semi-

formal representation of the conceptual definitions that is

as far as possible independent of particular symbol-levelencoding offers greater vantage in the design and

development of ontologies for information agents.

5. CONCLUSION

The formative evaluation of an ontology for information

agent encompasses validation and verification of the

ontology content to corroborate the reliability of the

ontology.

The validation of the ontology is necessary to ensure

that a `right' ontology is being built. For this, we propose

to validate the ontology against its reference frames.Common frames of references are requirement

specification, competency questions and, shared

knowledge and information resources in the domain of

discourse on which the development of the ontology was

based.

The verification of the ontology content is necessary

to ensure that the ontology is built `right'. For this, we

propose to check on the compliance of the ontology with

respect to a feasible set of ontology evaluation criteria,

namely consistency and coherence, completeness,

conciseness, extensibility and expansibility, competency,

expressiveness and minimal encoding bias.

At present, ontology evaluation is performed

manually. The question we ask is can suitable diagnostic

tools be developed to facilitate the formative evaluation of

ontologies. In addition to checking on the logical

correctness of the ontological definitions, it is interesting

to study to what extent completeness evaluation can be

automated. For instance how can we gather, formalise

and pose competency questions to ontologies using query-

answering tools in an organised manner? Conversely,

could we make it possible for ontology development tools

to systematically generate the competency questions that

it is capable of answering, which can be easily verified by

a knowledge engineer? Such supporting tools can largelyalleviate the burden of competency evaluation.

ACKNOWLEDGEMENT

The author expresses special thanks and gratitude to Prof.

Leon Sterling of The University of Melbourne for

inspiring the research on ontology and for his comments

on earlier writings on ontology design and evaluation.


6/6

REFERENCES

Boehm, B., 1981. Software Engineering Economics. Prentice-Hall. NJ.

Chandrasekaran, B and Josephson, J. R., 1999. What AreOntologies, and Why Do We Need Them? In IEEEIntelligent Systems, January/February 1999.

Fernandez, M., Gomez-Perez, A and Juristo, N., 1997.Methontology: From Ontological Art Towards OntologicalEngineering. In Proceedings of AAAI97 SpringSymposium on Ontological Engineering, StanfordUniversity, CA.

Fox, M and Gruninger, M., 1998. Enterprise Modelling. In AIMagazine, 19(3).

Gomez-Perez, A., 1996. A Framework to Verify KnowledgeSharing Technology. In Expert Systems with Application,11(4).

Gomez-Perez, A., 1999. Evaluation of Taxonomic Knowledgeon Ontologies and Knowledge-Based Systems. InProceedings of KAW99 Workshop on KnowledgeAcquisition for Knowledge-Based Systems, Alberta.

Gomez-Perez, A., 2001. Evaluation of Ontologies. InInternational Journal of Intelligent Systems, 16(3).

Grosso, W., Eriksson, H., Fergerson, R., Gennari, J., Tu, S andMusen, M. Knowledge Modeling at the Millennium (TheDesign and Evolution of Protege-2000). In Proceedings of

KAW99 Workshop on Knowledge Acquisition, Modellingand Management, Alberta.

Gruber, T., 1993. A Translation Approach to PortableOntologies. In Knowledge Acquisition, 5(2).

Gruber, T., 1995. Toward Principles for the Design ofOntologies Used for Knowledge Sharing. In InternationalJournal of Human Computer Studies, 43(5).

Gruninger, M and Fox, M., 1995. Methodology for the Designand Evaluation of Ontologies. In Proceedings of IJCAI95Workshop on Basic Ontological Issues in KnowledgeSharing, Montreal.

Moody, D and Shanks, S., 1994. What Makes a Good DataModel? Evaluating the Quality of Entity RelationshipModels. In Proceedings of ER94 Conference on Entity-Relationship Approach. Springer.

Staab, S., 2002. Trends and Controversies: Ontologies KISSESin Standardization. In IEEE Intelligent Systems,March/April 2002.

Uschold, M., 1996. Building Ontologies: Towards a UnifiedMethodology. In Proceedings of Expert Systems96Conference of the British Computer Society SpecialistGroup on Expert Systems, Cambridge.

Uschold, M and King, M., 1995. Towards a Methodology forBuilding Ontologies. In Proceedings of IJCAI95Workshop on Basic Ontological Issues in KnowledgeSharing, Montreal.

Formative Evaluation of Ontologies for Information Agents

Documents

Transcript of Formative Evaluation of Ontologies for Information Agents