APECKS: using and evaluating a tool for ontology construction with internal and external KA support

Int. J. Human-Computer Studies (2002) 56, 375–422doi:10.1006/ijhc.1000Available online at http://www.idealibrary.com.on

APECKS: using and evaluating a tool for ontologyconstructionwith internal and external KA support

Jeni Tennison

Dudley Court, Beeston, Nottingham NG9 3HZ, UK. email: [email protected]

Kieron O’Hara and Nigel ShadboltIntelligence, Agents and Multimedia Group, Department of Electronics and Computer

Science, University of Southampton, Highfield, Southampton SO17 1BJ, UK.

email: {kmo,nrs}@ecs.soten.ac.uk.

(Received 30 September 2001 and accepted in revised form 21 February 2002)

This paper describes Adaptive Presentation Environment for Collaboration KnowledgeStructuring (APECKS), an experimental tool for collaborative ontology construction.APECKS takes a different line to most ontology servers, in that it is designed for use bydomain experts, possibly in the absence of a knowledge engineer, and its aim is to fosterand support debate about domain ontologies. To that end, it does not enforce ideals ofconsistency or correctness, and instead allows different conceptualizations of a domainto coexist. The system architecture and life cycle are introduced, and three extensivescenarios are outlined, showing how APECKS supports ontology construction,learning, ontology comparison and discussion. APECKS has also been used by severalsubjects during an evaluation experiment, and the results of this experiment aredescribed. A particular factor about APECKS is that, as well as providing internal KAsupport, it is designed to interface with web-accessible KA tools, thereby allowingtheoretically unlimited KA support for users. The prototype used WebGrid-II asexternal KA support, and the issues involved in integrating APECKS and WebGrid arediscussed in detail.

# 2002 Elsevier Science Ltd. All rights reserved.

1. Introduction

In this paper, we wish to describe APECKS, an experimental tool for ontologyconstruction that exploits WebGrid as an external support for knowledge acquisition.Ontologies have been increasingly recognized as key resources for knowledgemanagement and engineering, for example as aids to the extraction of informationfrom natural language documents or web pages, or as a way of fixing the objects,concepts and relations in a domain in order to facilitate unambiguous communication.In particular, ontologies will become increasingly important as supporting methods forrealizing knowledge capture on the semantic web (Berners-Lee, Hendler & Lassila,2001).

1071-5819/02/$ - see front matter # 2002 Elsevier Science Ltd. All rights reserved.

J. TENNISON ET AL.376

In such a context, there is a need for tools for the construction and maintenance ofontologies. If organizations are to become what Smith and Farquhar (2000) callknowledge-powered enterprises, where knowledge management is ubiquitous andembedded in the organizational processes, and all employees participate in managingknowledge throughout a seamless information space, then demand will be less forknowledge engineers, and more for usable knowledge engineering functionalityincorporated into tools. This will therefore create a need for tools that are usable inthe absence of a knowledge engineer.

Adaptive Presentation Environment for Collaborative Knowledge Structuring(APECKS) is a tool designed to allow users to collaborate in the production ofontologies, maintaining their own personal ontological spaces and engaging indiscussions about their own and others’ ontologies in a domain (Tennison, 1999).Rationales for design decisions are kept, as are the threads of arguments, in order topromote a consensus where possible, and where decisions are contested, to allow otherusers to follow debates and make their own judgments accordingly. Although APECKSwas an experimental prototype, and has not been exploited by a large user base, we willargue in this paper that a number of interesting lessons have been learnt from itsdevelopment, evaluation and use, both for the practice of developing and maintainingontologies and about ontologies themselves.

The structure of the paper is as follows. We begin by discussing the issues involvedwith the development and maintenance of ontologies, and make a case for thecollaborative construction of ontologies. Next, we describe APECKS, showing how itmeets the requirements sketched in the previous section. We will then give threeextended scenarios of use of APECKS, and a sketch of its evaluation. In Section 5, wediscuss the use of WebGrid-II as an external support for KA within APECKS. Section6 discusses issues raised by the development of APECKS, including the future ofontologies and knowledge engineers in ‘‘knowledge-powered enterprises’’, lessons forevaluating KA tools, desiderata for ontology development environments on thesemantic web and the use of web-based tools as resources.

2. Ontologies

Ontologies, which have for some years been at the centre of a great deal of research inartificial intelligence, knowledge-based systems, natural language technologies and soon, are increasingly seen as a key technology for supporting the development andexploitation of the semantic web (Berners-Lee et al., 2001). In the current scramble forthe optimal languages and protocols for semantic web development, there is anadvantage for an ontology development environment that is adaptable for editingmodels in different web languages, of which the market leader is Prot!eeg!ee-2000 (Noy,Sintek, Decker, Crubezy, Fergerson & Musen, 2001), with its customizable output fileformat and user interface and extensible architecture.

In this section, we will review some of the issues surrounding the use, developmentand maintenance of ontologies in the context of the semantic web (of course, there areother contexts in which ontologies are as important, where the issues and emphasesmight be quite different). We will begin in Section 2.1, with a brief review of the central

APECKS: A TOOL FOR ONTOLOGY CONSTRUCTION 377

idea of an ontology server. In Section 2.2, we look at ontology maintenance, and theissues of responsibility, validation and consensus. In Section 2.3, we list some of thereasons for collaborative ontology development, and in Section 2.4, we discuss someattempts to create environments for collaboration.

2.1. ONTOLOGY SERVERS

Ontology servers are web-based resources that hold ontologies for general use. Theontologies are likely to be stored in a standard format, such as OIL, which has a frame-like look and feel, a formal semantics, and contains the requisite links to XML andRDF to ensure interoperability (Fensel, Horrocks, van Harmelen, Decker, Erdmann &Klein, 2000; Fensel, van Harmelen, Horrocks, McGuinness & Patel-Schneider, 2001).Whether or not the server representation uses a standard format, output shouldcertainly be in standard formats, to facilitate usability. Such a server will offer a view ofthe ontologies it stores through the web, usually in a frame-based representation.Hence, users will see web pages representing either individual objects or classes ofobject. A class or object’s page will give information about it, and links to other objectsand classes according to its position in the conceptual hierarchy. It is possible for anontology server to give different (possibly user-defined) viewpoints onto an ontology.So, for example, Co4 (Euzenat, 1996) allows users to define viewpoints that givealternative hierarchical classifications of the objects in the domain.

2.2. STRATEGIES FORONTOLOGY MAINTENANCE

Having placed an ontology onto a server, it is essential to maintain it, to keep it usable ina possibly fast-moving domain. In general, there are two strategies for ontologymaintenance, the centralized strategy and the distributed strategy. The virtues and vices ofeach are complementary. In a centralized ontology, there is a named person or body withfull responsibility for the content and structure, either of the ontology itself, or ofindividual modules. Anyone wishing to make a contribution to the ontology goes throughthis central authority. In a distributed ontology, people make additions as they see fit.

The person in charge of a centralized ontology can enforce quality and consistencyrules; however, the drawback is that new knowledge may take a long time to get addedas it is assessed and evaluated; particularly in fast-moving industries this may be aproblem. The difficulty is avoided in a distributed system, as anyone may makeadditions as soon as they are required. However, such a system would then be open tothe reverse problem, which is that the quality of every addition could not then beguaranteed.

It is, of course, possible to mitigate the effects of these difficulties. For instance, theSchlumberger knowledge hub discussed in detail in Smith and Farquhar (2000,pp. 23–29) has the so-called knowledge champions who take the responsibility forcertain modules of the knowledge base, and therefore this is centralized maintenance.However, a tiered approach to validation and QA allows people to contribute whenthey see fit. In the Schlumberger best practice ontology, employees can post ideas fordealing with problems. These would then be rated according to how far they had beenvalidated: a good idea is unproven, but has been placed on the corporate intranet for


colleagues to try out and test; a local good practice has been implemented and hasimproved results for a particular sector of the company; Schlumberger best practice hasbeen properly validated by the relevant knowledge champion, and determined to be thebest approach across the organization.

However, such an approach may only be possible within a particular organization;over an entire research community, across the web, such centralization, even with atiered approach, may be unrealistic. In the next subsection, we will discuss thealternative: the collaborative and remote construction of ontologies.

2.3. COLLABORATIVE ONTOLOGIES:WHY MIGHT WE NEED THEM?

When building an ontology, as we have seen, issues of correctness and consistency needto be addressed. In such circumstances, especially when knowledge engineers aresomewhat at the mercy of domain experts, it is easy to interpret ‘‘correctness’’ as‘‘consensual’’ (Farquhar, Fikes & Rice, 1997). Where consensus is weak, then onepotential application of knowledge engineering is to establish a common viewpoint, andof course there are many KE tools for the purpose (e.g. Shaw & Gaines, 1989; Boy,1996).

However, there are a number of reasons why such an interpretation of the knowledgeengineer’s task is not always desirable.

* It may simply be beyond the scope of even the most brilliant knowledge engineer toreconcile opposite viewpoints in a highly contested area. Furthermore, in a very fast-moving field, it may not be possible for any knowledge engineer to process opposingpoints of view and find consensus in real time if the knowledge itself is changing toorapidly.

* The cost/benefit structure of the knowledge engineering enterprise differs radicallybetween the cases, where there is a consensus (or at least consensus on a decisionprocedure), and when even the arbitration procedure is contested (Cowan, David &Foray, 1999). In particular, the effort involved in providing the expressive capacityfor a consensus in a contested field (i.e. producing a lingua franca on which all sidescan agree, and which can therefore act as the beginning of an understanding betweenall the parties, is a large cost and apt to render the whole enterprise uneconomic(Cowan & Foray, 1997).

* Much knowledge engineering activity is driven by a desire for archiving, theprovision/preservation of a corporate memory as opposed to automation. Hence, thedisparate views and arguments that can be found in a contested area may beprecisely what are required when the ontology is built.

* Such archiving efforts can be important initial steps in producing the expressivecapacity for consensus, thereby altering the cost/benefit structure of the knowledgeengineering enterprise. In other words, in a contested field, the cost structure willconsist of (1) a large initial (sunk) cost, as a result of developing the expressivecapacity, and (2) relatively small marginal costs of creating and maintaining aconsensual ontology (Cowan & Foray, 1997). The collaborative efforts we areimagining here would constitute the expenditure of the initial cost.

* Even if a consensual ontology is to be aimed for by a knowledge engineer, and thecosts of such an effort deemed not too great, it may be desirable for there to be an


initial period of KA from all the parties, and a medium which would provide for thecomparison of the different ontologies that may be produced. Hence the logging ofdisparate views would be the essential preliminary to criticism, comparison andconsensus.

* Even assuming that a knowledge engineer could moderate a consensus, ex hypothesi

in a contested field the participants themselves are unwilling or unable to produce aconsensus. Hence, if the general trend/requirement for knowledge engineers to bephased out of the knowledge engineering process [i.e. for knowledge engineering tobe ubiquitous, and performed by non-knowledge engineers (Smith & Farquhar,2000)] is to be supported, it is essential that some software support be provided forthe process of reaching consensus}it cannot be assumed that a knowledge engineerwill be available to moderate.

* If ontologies are required for web-scale applications, then it is unlikely that a singleontology, even for a relatively circumscribed domain, would scale up to do all thatsuch an ontology would be required to do, even if the heterogeneous web communitycould agree on one (cf. e.g. Fensel et al., 1999). Hence, methods for buildingontologies collaboratively, in parallel, and finding ways of relating disparateontologies are all likely to be important research efforts in the near future of thesemantic web.

* Last but not least, philosophically, there are decent arguments for supposing that theidea of a final or ultimate conceptualization of a domain is not going to appear, andthat knowledge and human interests are always going to be strongly connected,meaning that ontological relativity is a background condition that simply has to beaccepted (Dewey, 1929; Habermas, 1968/1987; Quine, 1969).

We might also note the importance of situated learning in today’s knowledgeeconomy, and the centrality to such learning of communities of practice (Wenger,1998). Many authors have already argued for the importance of ontologies to suchcommunities (Domingue, Motta, Buckingham Shum, Vargas-Vera, Kalfoglou & Farnes,2001), and the need to support reflective practice, particularly with the provision oftools to enable practitioners to articulate, critique and compare their informalunderstandings of a domain (Mulholland, Domingue, Zdrahal & Hatala, 2000).

Hence it is not always the case that the knowledge engineering process aims to createor moderate consensus, or that ontologies are necessarily tools for achieving consensus.We should also note that there are many other tools and approaches for achievingconsensus, and that knowledge engineering is not the only discipline concerned withthis issue. However, ontologies are, we suggest, a useful tool for exploring the nature ofconceptual differences}whether this exploration is intended to remove such differencesor simply to understand them}and the philosophy of APECKS is therefore to applycollaborative ontology technology to such domains.

2.4. COLLABORATIVE ONTOLOGIES

The issue that collaborative ontology servers need to address is the issue alluded toabove: how can the different contributions of the participants be moderated? Currentpractice varies from a free-for-all where anyone can make amendments to more


structured interactions. At the basic level, user read and write permissions can be usedto restrict access where desirable. There are a number of different ways that this can becarried out. In SHADE (Gruber, Tenenbaum & Weber, 1992), users subscribe to areasof interest, and are notified whenever changes to those areas occur. HyTropes (Euzenat,1996) allows users to annotate the pages it generates. Tadzebao (Zdrahal & Domingue,1998) supports argument between users on the ontology design, using text, GIF imagesand even hand-drawn sketches. It links with WebOnto (Zdrahal & Domingue, 1998), atool for supporting collaborative ontology browsing and editing using a directmanipulation interface, which incorporates the communication tools provided byTadzebao.

A different, standards- rather than communication-based, approach is applied by theOntosaurus browser (Swartout, Patil, Knight & Russ, 1996), which allows individualsto add to an ontology only when consistency is retained within the ontology as a whole,an approach made possible by the underlying representation language LOOM’sreasoning abilities and consistency checking. Of course, consistency checking is not onlyof use for collaboratively-built ontologies}it may be useful to check the output of asingle expert as well, especially over a long series of KA sessions.

Co4 (Euzenat, 1996) provides further functionality for collaboration, by structuringinteractions between the individual users. Groups of users who have reached aconsensus are associated with knowledge bases, i.e. the knowledge base that expressesthat consensus, and ontologies are built into these KBs. The KBs can then be placed ina hierarchy, depending on the consensual relationships of their users. Then, at thebottom of the hierarchy, users can have a number of KBs that they can alter as theyplease. If a change is proposed to the KB of a particular group, the group is asked forits opinion, and the change is accepted if and only if all the users in the group accept thechange. Individuals can also refuse to accept a change in their individual KBs, even ifthe group has made the change.

Hence, there are a number of different approaches to allowing collaboration duringthe building of ontologies, either allowing communication between the variouscollaborators, structuring the debate between collaborators or a mixture of both aswith Euzenat.

However, the fundamental claim we would like to make is that what is missed bythese approaches is that, when consensus is not achieved, the causes of thecommunication breakdown are not brought into focus. Often, the causes ofdisagreement remain tacit in discussions between experts}one of the vital functionsof a knowledge engineer is to make such tacit factors explicit. In the absence of aknowledge engineer, there is no more reason to think that a standard collaborativeontology server can bring them out, than there is to think that they will be brought outin the normal run of debate. Indeed, a standard ontology server, unmediated by aknowledge engineer, is basically an extension of normal debate using a new medium.

There are, of course, tools that are intended to capture the structure of debate, forexample the Issue Based Information System, IBIS (Conklin & Yakemovic, 1991);however, one problem with IBIS is that the discussion is based on a particular thread ofquestions. Questions, justifications and ideas developed in the context of a single pointof view, while they can make up an interesting debate, may raise issues that areorthogonal to the interests of others with radically opposed points of view. One of our


aims is to help isolate causes of communication breakdown in ways that make sensefrom every entrant’s point of view. ScholOnto (Motta, Buckingham Shum &Domingue, 2000) can be used to annotate or enrich digital documents in terms of theideas that are new, the conceptual foundations of the argument, and the impact atheory has had. ScholOnto uses an ontology to support the characterization of debate;this could be problematic in a contested context, in that the essence of epistemologicalcompetition is that viewpoints and perspectives differ, sometimes radically. TheScholOnto solution to this is to rely on a high-level ontology of scholarly discussionwhich keeps the claims about relations between arguments and ideas separate from thecontext and content of those ideas themselves. It may be that this distinction will behard to maintain in fields where debate is strong, in that the (perception of the) contentof ideas may often be influenced by perceptions of their relations with the hard core ofcompeting research programmes (Lakatos, 1970).

APECKS is intended to be brought to bear on disagreement, encouraging argumentand debate in order to uncover the root causes of the disagreement. The APECKSapproach seeks to enable experts to engage in structured debate, thereby addressing thesources of their disagreements}at least as far as ontologies are concerned}in theabsence of a knowledge engineer (although of course any server which will support suchstructured debate may be of value to knowledge engineers as well). In the next section,we describe the principles underlying APECKS, and then we will give three extendedscenarios of its use.

3. APECKS

3.1. PRINCIPLES UNDERLYING APECKS

APECKS is based around five principles.

* Personal ontologies.* Use of networked resources.* Support for KA techniques.* Recording of design rationales.* Structured communication between users.

3.1.1. Personal ontologies. The standard collaborative approaches to ontologies,discussed in Section 2.4, are aimed at creating an independent ontology. Suchontologies are in the main conservative, by which we mean that additions/alterations tothe ontology must be legitimized.

(1) In terms of the user. Special permissions are required for a user to change it. A usermust either subscribe to the server in some way or must otherwise apply for writepermission.

(2) In terms of the content. The additions/alterations must, for example, be consistentwith the current representations or must be approved by other users.

Such legitimization requirements, we argue, are counterproductive, especially (2),which is radically dependent on relatively extraneous factors such as who is currently


using the ontology, or the current state of the ontology. In other words, whoever getsthere first is at an advantage with respect to adding/altering the ontology, as he willhave fewer people with whom to argue, and less content in the ontology for him tocontradict.

In contrast, APECKS supports the creation of personal ontologies, owned byindividuals to represent their understanding of the domain. Such personal ontologiesare independent of others’ ontologies, although content can be copied in or adaptedfrom them. Furthermore, there is no requirement for consistency, either acrossontologies or within a particular ontology. Hence, personal ontologies in APECKS arenot conservative.

The aim of the repudiation of conservatism is to allow a dynamic process of creationand testing akin to brainstorming to take place during ontology construction.Divergence is encouraged in order to provide a no-holds-barred environment fordebate.

On a terminological point, many definitions of ‘‘ontology’’ stress the importance ofthe shared nature of the conceptualization (e.g. ‘‘an ontology is a formal, explicitspecification of a shared conceptualisation. . . . ‘Shared’ reflects the notion that anontology captures consensual knowledge, that is, it is not private to some individual,but accepted by a group’’ } Studer, Benjamins & Fensel, 1998; cf. also Duineveld,Stoter, Weiden, Kenepa & Benjamins, 2000). Clearly, personal ontologies as definedhere do not fall under such definitions. However, our view is that a better term than‘‘shared’’ here would be ‘‘sharable’’ (after all, an ontology created by a single developerdoes not suddenly ‘‘become’’ an ontology at the point when someone else looks at it);the aim of APECKS is to facilitate users in making their conceptualizations of adomain sharable, and also to introduce others to differing sharable conceptualizations,which may or may not have been drawn up by more than one person. There is no need,though, for us to get into the theology of ontology; the reader merely needs to bear inmind that, in the context of APECKS, a ‘‘personal ontology’’ generally developed by asingle person will look very like an ontology as usually understood in knowledgeengineering, and can be seen as a type of ontology or not depending on the reader’spreferences.

Note also that the development of personal ontologies (from now on we shall dropthe inverted commas) related to each other as accounts of some domain ameliorates tosome extent the problem of version control, which is a particular problem for themaintenance of genuinely shared ontologies that fall under the definition of Studer et al.(Klein & Fensel, 2001). The structure of personal ontologies in the domain, if resultingfrom changes to a central one, will look something like a derivation tree; however, eachparticipant in the ontology development programme will be responsible for a branch ofdevelopment. This makes the versioning problem somewhat more tractable; other usersmay make suggestions about a particular personal ontology, but that ontology willhave an owner who will make the final decision. Nevertheless, as it is possible, and evenencouraged, for users to build as many ontologies as they have hypotheses about thedomain that they wish to ‘‘try out’’, there is no doubt that version control could becometricky, and that users will have to be careful about ‘‘housekeeping’’.

Change is extremely important in systems like APECKS designed to supportevolutionary processes. Support is needed for managing changes, recording changes


and their rationales and allowing changes and rationales to be searched and browsed, aswe shall see in examples in Section 4.

3.1.2. Networked resources. Ontology servers, as their names suggest, are concernedwith serving information. However, APECKS is designed to act as both a server and aclient, in order to utilize resources from the Internet. The current version of APECKS islargely a proof of concept, and hence only uses a single networked resource, WebGrid(see Section 5). However, there is clearly a wide variety of resources that couldtheoretically be used, in particular content from other ontology servers. Ontologiesfrom other servers could be accessed by APECKS and held in the same way as thepersonal ontologies of Section 3.1.1. Such ontologies, depending on the structure of thediscussion within APECKS could be privileged or not, as users decided. Hence,APECKS could be used as a wide-ranging ontological construction and critiquingresource. The development of web-based learning and collaboration resources is a cleartrend (e.g. Eisenstadt & Vincent, 1998), and tools like APECKS can provide anindependent platform for integrating interesting sets of services.

3.1.3. KA techniques. In order to give support to non-knowledge engineers, APECKSwas designed to include some knowledge acquisition support for users. KA support cancome at two levels. First, APECKS itself includes some KA functionality of its own (seeSection 4.3). Second, APECKS, in the spirit of its intention to exploit networkedresources, can use web-accessible KA tools as they become available (discussed in detailin Section 5). The current version uses WebGrid (Gaines & Shaw, 1996) as part of itsKA repertoire.

3.1.4. Design rationales. Change is, of course, extremely important in the contextsenvisaged by APECKS. But change tracking in hypermedia is relatively problematic.When changes are made to APECKS ontologies, support is given not only forrecording the changes, but also for recording the rationale of the changes and makingthese rationales available explicitly and clearly to users. Users are informed, uponlogging in, of changes that have been made that impact on their personal ontologies,and annotations to the changes made available for searching.

In this way, APECKS supports an argumentative process of critique, in which theindividual user is in control. No-one has changes forced upon his personal ontology. Itmight be possible, in a future version of the system, to exploit principled supportmechanisms based on ontology properties, such as those in OntoClean as described in(Welty and Guarino, 2001, pp. 65–68). However, one of the governing principles ofAPECKS is that consistency is not forced upon users, and so such support would haveto be quite lightweight (see next subsection).

3.1.5. Structured communication. The main engine for debate in APECKS is thecomparison of personal ontologies. This is an important addition, as it is potentially away of breaking argumentative deadlock between contesting points of view. Putanother way, APECKS, in the absence of a moderating knowledge engineer, can act asa moderating influence itself, comparing ontologies and presenting neutral accounts ofthe differences and similarities for discussion.


The framework used by APECKS for comparison is the consensus/conflict/corres-pondence/contrast framework of Shaw and Gaines (1989). Hence, not only consistencybetween ontologies is tested, but also the degree of consistency.

Part of the KA capability of APECKS is to prompt users to respond to the results ofcomparisons. For example, if two users classified domain instances in the same way,while using different terms for their classifications, then they would be prompted byAPECKS either to change nomenclature to bring themselves into line, or to initiate adiscussion about why that could or should not be done. Such discussions are likely touncover richer domain content that would otherwise be available. So in the example, itis possible that a belated agreement could be reached on nomenclature, but it is alsoconceivable that there might be interesting relationships between the two methods ofclassification (e.g. one set of phenomena might cause the other set) which could beuncovered using such devices.

The prompting of the user by APECKS distinguishes its communication componentfrom interesting discussion space software such as the Digital Document DiscourseEnvironment (D3E: Buckingham Shum & Sumner, 1998), which focuses around asingle document}which may or may not be owned by the D3E discussion spaceowner}and generates a useful frame-based environment with automatic hyperlinkingand an integrated space for critiquing a document collaboratively. D3E is an enablerrather than a guide for the critique, and is focused around a single document, asopposed to APECKS which is driven by comparison of two ontologies which may havestructural similarities.

However that may be, APECKS will at a minimum force distinctions andassumptions out in the open. While they remain tacit, it is quite possible that theroles that such assumptions play in preventing consensus would go undiscovered.

3.2. SYSTEM DESCRIPTION

APECKS is a system for users, independent of a knowledge engineer, to attempt toreach a consensus by critiquing and coming to understand and maybe appreciatingopposing points of view. In this way it is similar to systems such as Co4 (Euzenat, 1996)and SHADE (Gruber et al., 1992), although it differs by focusing not on the endproduct (an ontology), but on the process of argument and debate that are involved inreaching a consensus.

To this end, APECKS uses a semi-formal knowledge representation and designrationale formalism, with affinities to systems such as IBIS (Conklin & Yakemovic,1991) or QOC (MacClean, Young, Bellotti & Moran, 1991). Such formalisms allow abreadth of expression sufficient for non-knowledge engineer users, yet providesufficient structure for APECKS itself to moderate the debate.

3.2.1. Technology. APECKS is based on a Multi-user domain Object Oriented; (MOO:Curtis, 1992), an Internet- and text-based environment designed for multi-userinteraction such as chat rooms and on-line events. The primary purpose for MOOswas to provide a communication facility that was richer than inter-relay chat (IRC) andless oriented towards adventure games than multi-user domains (MUD). MOOs are not


particularly conventional, but the underlying technology provides the followingadvantages for APECKS.

* Object-oriented databases. MOOs consist of an OO database that defines theindividual MOO and a server program which interprets and runs the database file.This object-oriented structure means that OO representations, such as knowledgeengineering frame representations, are very natural within it.

* Internet accessibility. MOOs are accessible via the Internet using the telnet protocol.The mechanisms for opening, closing and maintaining connections betweencomputers necessary for ontology servers are built into the MOO server program.Several MOO developers have built on top of this and made available MOOdatabases which allow objects within them to be viewed and manipulated throughthe WWW.

* Programmability. MOOs are programming environments in which objects can beprogrammed to interact with users or other objects. The programs are part of thedatabase itself, and hence do not require compiling, allowing rapid prototyping.And, with an eye to APECKS’ being a WWW server, MOOs can be programmed togenerate HTML dynamically.

* Multi-user synchronous communication. MOOs allow multiple users to connect tothem and interact with each other using synchronous text-based communication.Built into the standard LambdaMOO database (Curtis, 1992) are user representa-tions, which facilitate user modelling, and other communication methods, such asasynchronous mail.

APECKS was based on the standard LambdaMOO database with three amend-ments. Firstly, its database understands HTTP, allowing it to act both as a server and aclient. Secondly, its database generates vanilla HTML dynamically on the basis of itsinternal object representations, enabling the user to interact with its objects via a webbrowser. And thirdly, a number of generic objects were added to the database, as thebasis of APECKS’ knowledge representation schema (Section 3.2.2).

3.2.2. Knowledge representation schema. APECKS uses a frame-based schema basedon the Frame Ontology (Karp & Gruber, 1997). As the Frame Ontology is the basis ofknowledge representation in Ontolingua, it brings APECKS into the Ontolinguacommunity, enabling, for instance, the use of Ontolingua and the Generic FrameProtocol for facilitation of translation between APECKS representations and those ofother frame representation systems. And of course, given the APECKS philosophy ofthe use of net-based resources, it means that APECKS can read in any ontologyavailable on the Internet that has been either developed using Ontolingua, ortranslatable with it.

Hence, APECKS knowledge representation looks familiar: individuals are groupedinto classes, and have slots which define values and facets. The main inference allowedis inheritance; multiple inheritance is handled through the class hierarchy.

The class hierarchy is currently defined not by users, but by the relationships betweenclasses and individuals}a restriction designed to increase the comprehensibility of theschema for non-knowledge engineers, and users not au fait with the standardfunctionality of knowledge representation formalisms. This is, of course, fairly


non-standard; the rationale behind it is to restrict the expressivity of the system from thepoint of view of the user, in order to cut down on opportunities for users inexperienced inlogic or knowledge engineering concepts to create ‘‘needlessly’’ inconsistent ontologies. Afuller evaluation of the system, it should be noted, needs to be done to collect and assessevidence that the envisaged users would benefit from such restrictions.

On a wider scale, knowledge is held in APECKS in a number of domains, and eachpersonal ontology is associated with a particular domain. Hence, each personalontology can be seen as a particular view of the constituents of a domain, and they arerelated to each other precisely by being of the same domain. Each ontology is definedby a single user, and users can define as many ontologies as they like, to describeparticular aspects of the domain, to try out different conceptualizations, or toinvestigate the requirements of different tasks within the domain. For this reason (andto avoid the terminological problems alluded to in Section 3.1.1}cf. Studer et al.,1998), personal ontologies are termed roles in APECKS. Roles are never shared withinAPECKS, but are always held independent of each other.

However, one would expect the same individuals and classes to turn up often indifferent roles}i.e. they may share content. This is to be expected, but is enshrined byAPECKS allowing (parts of) ontologies to be cut and pasted into others. Identity ofindividuals is conferred by slots for information to define it uniquely, such as its nameand description (which may be a simplifying assumption in the context of a number ofdebates), but, even when such individuals are held in more than one role, its classmemberships, slot values, etc. will be allowed to vary across roles. A description of howroles vary in their representation of the relations governing individuals is called acomparison.

Finally, there are meta-objects in each role: criteria are short assertions about arepresentation and the reasons for its having the form it does, while annotations arelonger, more discursive accounts which may take the form of a dialogue or argumentabout the form a representation has taken. Criteria and annotations, of course, areessential for the requirement on APECKS to support design rationales and structuredcommunication.

3.2.3. Interface. Each object within APECKS}each individual, class, slot, role,domain, comparison, criterion and annotation}has web-pages associated with it,dynamically created on the basis of the current state of the object and the user who istrying to access it. Each object can be viewed in a number of different modes.

* Main view. Gives the essential information about an object, its name, description,and then important information such as slot values, slot constraints, etc.

* Changes view. Gives the changes that have occurred to the object since the last timethis page was accessed. It also supplies access to a search engine to look specificallyfor changes, e.g. based on who made the change.

* Comments view. Lists the annotations and criteria relevant to the object.* Help view. As well as the standard help, gives users a list of possible options for the

next move, an important aid in unassisted KA.

Figure 1 shows the page associated with the main view of an individual. The pagebrings together material for the individual Granite from an ontology of geology,


including a picture of a sample and some material from a knowledge acquisitionsession; this content may be copied in from a third party, or may be created anew.Having been created with APECKS, this content can be copied by other users, orcommented on or annotated by other users or the creator of the page. The buttons atthe top of the window support the navigation through the different views of the Graniteindividual.

Figure 1. The APECKS interface for an individual.


4. APECKS in use

In this section, we will give a number of scenarios for the use of APECKS. We begin byshowing how APECKS’ life cycle has been conceived. Then we look at three aspects ofthe use of APECKS: as an ontology server, a learning tool and a research tool. We closethe section with a brief account of a small-scale experiment to evaluate APECKS.

4.1. APECKS’ LIFECYCLE

The APECKS life cycle has four phases. We will briefly describe them here; we will seeconcrete examples in the rest of this section.

4.1.1. Seeding. To begin with, APECKS needs to ‘‘seed’’ a domain with knowledge.Information is given to APECKS to start the collaborative process off. As Smith andFarquhar (2000, p. 28) put it in a slightly different context, ‘‘it is difficult if notimpossible to convince community members to contribute to an empty shell. . . . Notonly must there be content from the launch date, but it must be quality content as well.’’Individuals in the domain are given names, descriptions and URLs of images ifappropriate. More can be created later by users; the aim is simply to ‘‘get the ballrolling’’.

There are two steps to the seeding process. The first is to generate the user identitiesof the initial group of people who will be working on the domain; it is, of course,essential in collaborative systems that persistent user identities allow collaborators toknow to whom they are talking. The second step is to specify the domain in enoughdetail for users to understand what it is that they are discussing. Such specificationmight include an informal description of the domain and a number of individuals(specified in as neutral a way as possible, with their names, short descriptions, pictures)and criteria against which personal ontologies can be judged.

The aim is to produce an uncontroversial picture of the domain to act as an informalscoping document for the users (in a highly contested domain, of course, it may beimpossible to avoid controversy even at this early stage). However, the dialectic of thediscussion may well result in this early attempt at scoping being transcended.

4.1.2. Ontology construction. Ontology construction involves users constructing a roleto represent their personal view (maybe hypothetically) of the domain. This mightinvolve creating new individuals, slots, etc, or grouping them differently. The user has afree hand to document his view of the domain.

KA support is given by APECKS for this process. Internally, APECKS provides helppages that list the possible actions for a user at any stage. In general, there are threetypes of action that the user can pursue.

* Expand the ontology by adding individuals and/or classes.* Rerepresent the ontology, e.g. by re-drawing class boundaries.* Maintain or re-establish consistency.

In addition, the user can}is encouraged to}annotate the changes he makes.


These are, of course, relatively basic operations. In a standard ontology developmentmethodology, such as METHONTOLOGY (Fernandez, G !oomez P!eerez, Pazos & Pazos,1999), or others reviewed in G !oomez P!eerez & Benjamins (1999; Section 2.2), theknowledge level construction of ontologies is facilitated by the identification of variousontology construction tasks, and the imposition of some sort of life cycle over this,dictating to some extent when the various construction steps are to be taken. However,to a large extent, the ontologies developed using APECKS will be constructed by userswithout knowledge engineering experience, for whom ontology development is a newdeparture. For that reason, only the simplest steps, to create fairly vanilla-flavouredontologies, are available, and the user is given the small menu of actions discussedabove. A number of actions are likely to be made in response to (unpredictable)comments and questions, rather than being positively guided by a methodology. In thiscontext, this should not be harmful, cf. (G !oomez P!eerez & Benjamins, 1999): ‘‘Ifontologies are built on a small scale, some activities can be skipped.’’ For usersexperienced in knowledge engineering, of course, there is no reason why they should notfollow their favourite methodology. It may also be, in a properly evaluated andmarketed version of the tool, some methodologically-grounded environment such asOnto-Edit could be incorporated into the system.

As noted earlier, APECKS is also designed to allow web-accessible KA support ofany kind to be used. Currently, WebGrid is the exemplar of this eclectic approach. Wewill see the use of WebGrid in the scenarios below.

4.1.3. Ontology comparison. Constructed ontologies can be compared with each other,using the Shaw and Gaines (1989) consensus/conflict/correspondence/contrast frame-work, which has had to be adapted to work in ontologies (as opposed to the repertorygrids for which it was designed). There are important differences between intra-ontology relationships and intra-grid relations (see Section 5.1.1 for more detaileddiscussion on this). First, some slots defined within an ontology may not holdinformation that is suitable for comparison (for instance, information that is designedto identify an object uniquely, such as its name). And second, elements of ontologies aredefined not only by their slot values, but also by their positioning with respect to theclass structure.

APECKS supports the comparison of the class structures of two ontologies in two ways.

(1) Recoding the class structure in terms of categorical slots on the individuals in therole, or Boolean rating scales recording whether an individual is a member of aclass.

(2) Comparing classes in terms of their individual members. This allows comparisonsbetween roles to be made on the terms set out in Table 1.

APECKS does not include facilities for ontology merging (producing a coherentontology including all the information in the two source ontologies) or alignment(making the source ontologies consistent and mutually coherent while keeping themseparate). It does, however, produce a list of suggestions for actions that will produce orpreserve consistency between two ontologies. Hence, it could act as a (rather tedious)merging tool, rather more in the spirit of PROMPT (Noy & Musen, 2000), which

Table 1Reinterpreting the Shaw/Gaines categories for APECKS comparisons

Shaw/Gaines category Criterion for an APECKS comparison

Consensus There are classes in each role with the same name and the samemembers

Conflict There are classes in each role with the same name and yet differentmembers

Correspondence There are classes in each role with the same members and yetdifferent names

Contrast There is no overlap in either terminology or class structure


performs some merging/alignment tasks automatically and guides the user, where hisintervention is required, than in that of Chimaera (McGuinness, Fikes, Rice & Wilder,2000), which instead provides analysis or guidance for the user at any stage during themerging process, but which is restricted to pointing out problems in the mergedontology and leaves the decision of how to remedy the situation to the user.

The changes that PROMPT, for example, would suggest here are much morethoroughgoing than those that APECKS would suggest on the basis of its comparison.For instance, in an example taken from Noy and Musen (2000), a merging of twoclasses will prompt PROMPT to ensure that the slot structure, subclass structure andsuperclass structure retains the structure of each original source, as well as ensuringconsistency and coherence, which is no small task; such a merging is likely to requireuser intervention to sort out reference conflicts, import classes from elsewhere, or makefurther merges. APECKS’ suggestions will be much more conservative. However, asAPECKS does not assume that the user is unhappy with a situation where his ontologyis inconsistent with someone else’s, it does not carry the assumption that there is anyneed to carry out the vast list of difficult actions to merge ontologies; one of the actionsthat APECKS is likely to suggest when presented with an inconsistency between twoontologies is for the user simply to add an annotation explaining why he prefers it hisway (Section 4.1.4).

We can see examples of APECKS being used for discussion in Section 4.4. As wehave noted (Section 3.1.3), as APECKS is designed to be able to use networkedresources, there is no reason why a user should not be able to exploit a web-accessibleversion of PROMPT or Chimaera, say, if he so desired, or alternative ontology-comparison methods such as those reported in Maedche and Staab (2001), whichexamine the overlap between ontologies considered as semiotic systems at severalsignificant levels. It may be that such sophisticated methods are, at least initially,overkill, given that the intended user of APECKS is unlikely to be versed in logic orknowledge engineering, and that the ontologies created may be relatively primitive inthe first instance, or relatively trivial variants of the ‘‘seed’’ ontology; the open-endedstructure of APECKS lets the user decide.

4.1.4. Discussing ontologies. APECKS supports a range of styles of collaboration, andencourages discussion whenever a comparison of roles has been made. Thecommunication can be either synchronous or asynchronous, with users arguing the


case for their own design decisions with owners of other roles and other interestedparties. In this idea of promoting discussion to aid communication between people withpossibly different views of their domains, the philosophy of APECKS is similar to thatof Compendium (Selvin, Buckingham Shum, Sierhuis, Conklin, Zimmerman, Palus,Drath, Horth, Domingue, Motta & Li, 2001), except that Compendium is intended tofocus primarily on the face-to-face meeting, while communication in APECKS ismediated, to some extent at least, by the text-based MOO technology.

Synchronous communication is supported through the standard functionality of theunderlying MOO foundation of APECKS}i.e. it is already a vehicle for synchronousdiscussion between users. Discussions in virtual rooms between users can be associatedas annotations to the objects under discussion. Freeform annotation of objects,including the output of synchronous discussions, can go on at any time, and hence userscan catch up asynchronously with debates.

All of this communication is unstructured, but APECKS also supports the use of theQuestions, Options, Criteria (QOC) methodology (MacClean et al., 1991). In this way,common argumentative/communicative actions can be formalized, and the system canprompt for responses from users directly. In addition, structure is useful in contesteddomains when the standard patterns of debate have not yielded a consensus.

4.1.5. Connections between the phases. The connection between these four phases is notdictated by the system. Nevertheless, there are obvious priorities. Seeding is an initialphase, to begin the whole process. Ontology construction should precede comparison inthe first instance, and discussion is likely to follow the first comparison. Thereafter anyor all of the final three phases can happen, as different users have different priorities(and of course, different users may be doing different things at the same time; one usermay be amending his ontology while another is responding to annotations).

4.2. APECKS AS AN ONTOLOGY SERVER

The most basic function of APECKS is as an ontology server. In this scenario, we willfollow a knowledge engineer as she populates an ontology, and show how APECKSperforms the ontology server function of allowing two knowledge engineers to share anontology. Hence, for this scenario, we do not need to consider knowledge acquisition orontology construction techniques for users unversed in the subtleties of knowledgeengineering.

Dr Z and Dr Y are two knowledge engineers. Dr Z is entering the Sisyphus IIIexperiment (Shadbolt, O’Hara & Crow, 1999), and is beginning by developingan ontology of the domain. The initial step within APECKS is to create a domain(Figure 2). It is the aim of Dr Z to create an ontology for the geological domain ofSisyphus III, and then to share that ontology with Dr Y, who will take over for the laterstages of the experiment, thereby demonstrating the reusability of ontologies. In thisscenario, we follow Dr Z’s ontology construction, and handover to Dr Y.

Seeding the ontology can be done from any appropriate source, or by hand. Dr Z canuse the description of the Sisyphus III experiment itself to fill in the description of thenew domain (Figure 2). She can then populate this domain by creating individuals forthe various rocks discussed in the first phase, and for the various minerals discussed in


basic texts on igneous rocks, and use materials to hand to provide the descriptions ofthese objects (we have already seen such a page in Figure 1).

This completes the seeding of the domain, and Dr Z now moves onto theconstruction of an ontology proper. The information provided so far has been generalpurpose material supplied by a third party. Dr Z can now create a role for her ownresearch group, and given her experience as a knowledge engineer, she does not use theAPECKS KA prompts}we can see use of these in Section 4.3.

APECKS allows a number of class formation operations. Individuals can be dividedinto classes. More interestingly, if a set of classes is mutually exclusive, they can be

Figure 2. A new domain for Dr Z’s Sisyphus III experiment, including a description of the domain.


marked as a partition of a subclass (Figure 3). They can be marked ‘‘exhaustive’’ if theycontain between them every member of the superclass.

In Figure 3, Dr Z has named a partition ‘‘grainsize’’, and divides the instances of‘‘rock’’ between ‘‘coarse grain’’, ‘‘medium grain’’ and ‘‘fine grain’’. The fact that there isa partition rather than a mere division into classes is shown in Figure 4. At the top wehave the window showing a straightforward class division, while at the bottom, thechange to an exhaustive partition has been registered; the name of the subclass divisionappears next to the names of the subclasses in it, while the exhaustiveness is symbolizedby an asterisk.

Classification of instances is done using a window such as Figure 5. In this window,instances can be sorted with respect to classes by clicking on the buttons to the rightof the name of each instance. If Dr Z is unsure of the classification of a particularinstance, or none of the classes applies, she can ordinarily leave it blank. Sometimes}as in the example in Figure 5}the classes are known to form an exhaustive partition,in which case APECKS advises the user to complete the form without leaving anyblanks at all.

The same information can, of course, be encoded as slot values. This is an ontologydesign decision, and redundancy can be built into the system by encoding theinformation both ways. Slots can be related to other slots as inverses, ranges andconstraints, defaults, cardinalities, etc. can be set. Slot creation can be seen in thescreensnaps in Figure 6; the upper window shows the slot details being created,including the three values the slot can take, while the lower window shows thecardinality being set. Having created the slots, default values can be linked to theoriginal partition, so that members of a particular subclass of the partition wouldinherit the relevant slot value.

Figure 3. Defining a new partition.

Figure 4. The move from a subclass division to a partition.


While an ontology is being created, rationales can be kept alongside the designdecisions. So, for example, when Dr Y creates his own role, using Dr Z’s ontology asthe core, he merely makes a copy of Dr Z’s role. Then, if he adds new instances forexample, these can be annotated. Figure 7 shows how changes to an ontology aredisplayed, while Figure 8 shows how such changes might be annotated. These

Figure 5. Classification window.


annotations might become the basis for a discussion between Dr Z and Dr Y (and otherinterested parties), as will be shown in later scenarios.

In this scenario we have seen how an ontology could be built up from seed material,and copied among different users, all of whom retain ownership of their own particularunderstanding and view of the domain in their own roles. In the next scenario, weexamine the use of APECKS to construct ontologies by users who may not be au fait

with standard knowledge engineering techniques, and hence we will see the use ofAPECKS’ KA support.

4.3. APECKS AS A LEARNING TOOL

In this scenario, we imagine Miss X, a geology student who wishes to use APECKS as ameans of revising her knowledge of igneous rocks. She copies Dr Z’s ontology into anew role which she owns. However, she need only select and copy the individuals shewants for her role; she is not constrained by Dr Z’s choice at all.

The basic KA support that APECKS provides is a window specifying which actionsare available for Miss X to perform. Figure 9 shows this window; note the additionalhelp support given by the available choices. Miss X is here choosing to create a newclass, on the suggestion of the system itself, which has spotted that none of theindividuals in her role currently lies within a class. If she then, for example, creates aclass containing a large number of objects, asking for available actions will elicit theadvice to create a number of subclasses of the large class, in order to provide a finer-grained structure.

Figure 6. Windows for creating slots.


Figure 7. Window showing changes to an ontology.


Further KA help is given when she wishes to classify the instances of her ontology.We have already seen in Figure 5 how the classification window prompts for instancesto be placed in classes. Miss X is not obliged to place all instances in classes, where sheis unsure or the decision is inappropriate (i.e. she can leave some of the switches blank),except in the event of an exhaustive partition, where they must be filled in. This is anexample of a way in which APECKS will prompt the user to find information that shemay not have been aware was required. Note also that the basic structure of theclassification window in Figure 5 is equivalent to a card sort tool (Major, 1992), thoughof course minus the card metaphor interface.

Miss X is, of course, at liberty to ignore all this advice if she so wishes. If she does,then she will be prompted for an explanation (which again she need not provide).However, if she provides an annotation as recommended, these annotations will act as acumulative explanation of the design of her ontology. Figure 10 shows the windows

Figure 8. Window showing annotations to changes.


relevant to annotating ignored prompts. The top window shows APECKS making asuggestion about the division of Miss X’s instances into two classes, light and dark as inFigure 5; it might also be done by creating categorical slots. Miss X doesn’t like thisidea}she prefers to think of the domain in terms of two different groups rather thanone large one, and expresses this in an annotation shown in the lower window.

We have seen in Figure 9 the window showing the actions available, and calling upthis window once more, Miss X might be tempted by the option of analysis. This optionwill call up a session with WebGrid-II.

Figure 9. Window showing actions available within APECKS.


WebGrid, thanks to its web-accessibility, provides APECKS users with the capabilityto perform repertory grid analysis of their roles. The issues involved in theincorporation of WebGrid’s functionality within the APECKS environment arediscussed in detail in Section 5. It is intended that the distinction between APECKSand WebGrid be hidden from the user, and we can see a page of Miss X’s interactionwith WebGrid in Figure 11. The interface guides Miss X through what of course maybe a very complex analysis of her role. The questions and suggestions made duringher interaction with WebGrid, in this scenario, have resulted in the elicitation ofa couple more constructs, the size of grains within a rock and the amount of quartz itcontains.

When Miss X clicks on the name of her role, the interaction with WebGrid is over,though of course she is unaware of this. At this point, it may be that a construct she hascreated within WebGrid has been translated into a slot but not given a name byAPECKS. At that point she will be prompted for a name, and she can give a descriptionas appropriate.

Figure 10. Annotating ignored prompts.


Figure 11. The use of WebGrid within APECKS.


It may be that some slots have limited ranges within various classes. For example,Miss X’s classes ‘‘light’’ and ‘‘dark’’ are related to her new slot ‘‘quartz content’’, inthat, by and large, light rocks have a higher quartz content than dark ones. She canrepresent this by restricting the possible slot values for each class using a form like theone in Figure 12. However, if she does this, then it becomes possible that inconsistenciescan arise.

As we noted in Section 3.1.1, APECKS does not require roles to be consistent,remembering its principle to act as a ‘‘brainstorming service’’. However, this does not

Figure 12. Restricting values for slots.


mean that APECKS provides no support for consistency; after all, that users are notforced to be consistent does not mean that they will never want to be. In particular, onecan certainly imagine the process of ontology construction beginning with a messy,inconsistent sketch of various candidate class structures, which is tightened up andgiven increasingly rigorous consistency requirements as the construction processcontinues.


In this spirit, APECKS can prompt users with changes to restore consistency to theirroles. Furthermore, there are three other mechanisms for preserving consistency.

(1) Partitions, once declared, are kept as partitions. This means that the intersection ofno pair of partition subclasses can be populated, and that the union of all thepartition subclasses is a subset of the superclass. In addition, when the partition isdeclared as exhaustive, the union of the partition subclasses must be equal to thesuperclass.

(2) The class hierarchy is generated from the class membership of individuals (X is asubclass of Y iff every instance of X is an instance of Y). Altering the classificationof individuals will automatically alter the class hierarchy.

(3) As we have seen above, slot values can be limited. Constraints can be put on slots.Ranges of values that slots can take can be specified; when an individual inherits aslot from a class, then it can only have a value (or range) for that slot that is withinthe range it has inherited (the exception to this being if the range for the slot in theclass is declared after the value for the individual). The user can also declare suchthings as ‘‘must have’’ values for a slot, inverses or logical constraints (e.g.symmetry, reflexivity) on slots. These can be violated by users if they wish(consistency is not enforced here), but the user will be prompted with theinconsistency should one arise. APECKS will suggest values to remove theinconsistency.

In the scenario we are considering here, Miss X, having specified a range for a slot,may come across an inconsistency. We can see what happens in the series of snaps inFigure 13. Window (a) points out an inconsistency. One of the rocks, ‘‘Andesite’’, hasan inconsistent value for ‘‘quartz content’’. She clicks on the question mark icon todiscover what the problem is, and is told [window (b)] that ‘‘Andesite’’ has been classedas a light rock (and therefore should have high quartz content), but has been givena low value for quartz content. We saw ‘‘Andesite’’ being classed as a light rock inFigure 5, while during the interaction with WebGrid-II, it was given values of ‘‘1’’ and‘‘low quartz’’ for the slot ‘‘quartz content’’. But the slot value restrictions defined inFigure 12 require light rocks to have a value of between ‘‘3’’ and ‘‘5’’ for the ‘‘quartzcontent’’ slot. Hence, an inconsistency. Miss X decides she was wrong, and in window(c) makes the change to the value of the ‘‘Andesite’’ ‘‘quartz content’’ slot.

Having created her ontology, Miss X can compare her ontology with others in thedomain, using APECKS’ comparison techniques. This may tell her, for instance, thatthe classification of rocks as ‘‘mesocratic’’ and ‘‘leucocratic’’ corresponds in large partto her classification of them as ‘‘light’’ and ‘‘dark’’, or may give her classifications ofinstances she was unable to complete. Hence, she may learn new facts about orterminology in the domain. We will now move on to a scenario where the APECKScomparison techniques are demonstrated in action.

4.4. APECKS AS A RESEARCH TOOL

To see APECKS as a research tool, with its comparison support, we can follow througha scenario using an ontology of mammals (this ontology was used in the evaluation ofAPECKS}see Section 4.5; the roles created during this scenario were used as ‘‘goldstandards’’ in that evaluation).

Figure 13. Consistency maintenance in APECKS.


We can imagine Professor W as a historian of science, who wishes to track the historyof mammalian taxonomies, and wants a system that will allow him to study theirdiachronic development. He wants to compare taxonomies from different authorities,to track their changes over time.

He will, of course, begin by creating a domain ‘‘mammals’’ and seeding it withindividuals and a few classes. He can then create multiple roles}recall APECKSsupports this}one for each taxonomy he wishes to study (he names them after theirauthors). He can then begin to classify mammals, and develop his own ontology, usingthe techniques we have seen in Sections 4.2 and 4.3. Having completed this setting up ofthe situation, he can begin to use APECKS to compare the taxonomies. Of course, hedoes not want to use the comparisons to change the taxonomies in this case, where


historical study is the focus of the effort. But he can use the prompts and hints given byAPECKS to uncover differences and to annotate the taxonomies he has.

Recall that APECKS uses Shaw & Gaines’ consensus/conflict/correspondence/contrast framework, summarized in Table 1. We discussed the issues raised in using thisframework in the context of APECKS in Section 4.1.3, including the relative simplicityof this framework as compared to, say, PROMPT (Noy & Musen, 2000)}a simplicitywhich is consistent with the APECKS philosophy (though a merging or alignment toolcould certainly be used with APECKS if the user required such support). Figure 14shows a number of comparisons that APECKS makes in this case.

(a) A summary of the different mammals classified by Simpson (1945) and Lacepede atthe end of the 18th Century.

(b) Extracts from the list of consensual classes from the comparison of Simpson (1945)with Corbet and Hill (1991).

(c) Extract from the list of conflicting classes from this comparison. Post-1945 analytictechniques have revealed that the Rufous elephant shrew is not related to the WestEuropean hedgehog despite their similarities, and that the common tree shrew isnot a primate.

(d) Extracts from the list of contrasting classes from this comparison. Simpson usedhigh-level distinctions between Theria, animals with wombs, and Eutheria, thosewithout, while Corbet and Hill focused on families.

The next phase of the comparison would be to use WebGrid’s sociogrid comparisons.Figure 15 shows the result of such a comparison between Lacepede’s and Simpson’sontologies. If we extract the comparison itself (Figure 16), we can see that Prof Wdiscovers high correspondence between their classifications of sheep and horses, butlow correspondence in the case of marine mammals. Given this information about therelationship between Lacepede and Simpson, Prof W can now annotate the rolesaccordingly.

Now, Prof W can advertise the domain to his students as a learning resource. Theroles give an overview of the mammalian taxonomies that have existed over time, whilethe annotations give the diachronic comparison, as well as other information that ProfW thinks would be useful. By attaching keywords to the annotations, Prof W canensure that the domain becomes a searchable archive of information, and he canmonitor his students’ own contributions by searching for any annotations they haveadded (Figure 17).

4.5. EVALUATION OF APECKS

To see how genuine users coped with APECKS, the tool was evaluated in a small-scalescenario using the mammal domain. Twelve experimental subjects were required to useAPECKS to collect and structure information for one of the three purposes.

* A computer-based guide for mammalian information (morphology).* A computer-based guide for zoos to decide what to feed mammals in their care (diet).* A computer-based guide for tourists, explaining what mammals they could see on

their travels (habitat).

Figure 14. APECKS as a comparison tool.


Figure 15. Page showing a sociogrid comparison.


The subjects were introduced to knowledge engineering concepts and terms in a veryrudimentary way, and then were required to use APECKS unaided (having been‘‘walked through’’ the use of the system by an experimenter for about 20min). Thesubjects were set tasks, intended to bring out particular aspects of the system and uponcompletion they were asked to fill in forms describing the usability of the system. Theseforms required the subjects to rate a series of activities on a seven-point Likert-like scaleranging from 1 (extremely difficult) to 7 (extremely easy); the activities to be evaluatedwere browsing, adding, changing and removing information, comparing roles anddiscussing information. The forms also allowed subjects space to make textualcomments.

The subjects were also informed that other subjects were working on the experiment,and they were encouraged to look at and discuss others’ roles. The system logged theAPECKS pages that were visited, together with the time spent at each. The states of thesubjects’ ontologies were also logged, and the ontologies evaluated by a knowledgeengineer at the end of the experiment. In all, the subjects had four sessions of usingAPECKS.

In this paper, we do not intend to go into great detail about the experimental results,but they were encouraging. Given the small sample size of the experiment, theconclusions that can be drawn are necessarily not as firm as desirable, but thedifficulties of evaluating KA software have been discussed elsewhere (Shadbolt et al.,1999), though see Duineveld et al. (2000) for a useful framework for evaluatingontological engineering tools. Interesting results were achieved respecting the system’susability, the quality of the subjects’ ontologies and the analyses of the subjects’

Figure 16. Searching the annotations.


navigation through the domain. In the remainder of this section, we discuss theimplications of the evaluation for APECKS; in Section 6.3, we will extrapolate fromthese results to extract some general lessons for evaluation projects.

4.5.1. System usability. The analysis of the reported usability over the four sessionsshowed that subjects found the system easier to use over time. It seems that subjectsfound APECKS particularly easier to use after the third session, after they had beenusing it for around 1.5–2 h. The only activities that did not show evidence of gettingeasier over time were removing information and having discussions.

The reported usability of the system during the first session was not significantlycorrelated with the usability in the other sessions. This indicated that those who foundthe system hard to use at first did not necessarily find it hard to use later, and vice versa.This may be due to the differing tasks during the first session compared to the laterones.

The degree to which the subjects found APECKS easy to use did not generallydepend on their previous experience with computers. Traceable effects of suchexperience disappeared in later sessions.

The comments given by the subjects were very helpful in identifying areas in whichAPECKS could be improved. The problems the subjects encountered fell into threemain areas: presentation, where subjects had problems with terminology, layout andtext size; navigation, where subjects got lost easily, being unable to get where theywanted to go; and discussion, where subjects often found themselves with nothing toreply to, and were reluctant to criticize others’ roles directly.

Figure 17. Extract from WebGrid comparison.


4.5.2. Ontology quality. The subjects’ roles had significantly more individuals, classes,hierarchies and annotations in the final session than they did in the first session,showing that the subjects continued adding information throughout the study. Inparticular, they used comparisons with other roles to select individuals and classes toadd to their own roles. They also added annotations throughout the study. The numberof class hierarchies within the roles increased over the sessions, while the number ofpartitions did not, suggesting that the subjects were creating classes as they came tomind, perhaps as a result of looking at comparisons with other people’s roles, ratherthan only using the prompts to create subclass partitions as they did in the earlysessions. The fact that extra slots and criteria were not created throughout the studysuggests that the subjects did not understand how to use them, or found their use toocomplicated in the earlier sessions, and were put off by them.


Qualitative scales on which the seed and final subjects’ roles were judged were used toaddress the important qualities of ontologies and distinguish between the seedontologies and those constructed by the subjects. Quantitatively, the ‘‘better’’ roles (asjudged qualitatively) tended to have both more individuals and more subclassificationsof those individuals. The number of annotations made on a role seemed to decrease theknowledge engineer’s perception of its qualitative goodness. The relationship betweenannotations and decreased qualitative goodness may be an artefact of the fact thatfewer annotations occurred on the (otherwise better) seed ontologies.

For the most part (one exception), the experimental design failed to generate biases inthe ontologies produced by the subjects according to the tasks they were asked to carryout. It may be that the tasks were too closely related (they were all classification tasks)to give any difference, but this is interesting considering the amount of discussion thatsurrounds the task-specificity of ontologies (e.g. Chandrasekaran, Josephson &Benjamins, 1998).

4.5.3. Protocol analysis. Unsurprisingly, subjects spent longer on pages when theyfilled in forms (WebGrid-II pages included) and on pages that contained prompts. Onboth these kinds of pages, there is a large amount of information to take in and acertain amount of consideration must be taken before proceeding to the next page:think-time dominates load-time. Types of objects that generate more information andmore forms, such as annotations and comparisons, were thus visited for longer thanthose that do not, such as the domain and individuals.

The average amount of time spent per page did not change over the four sessions, nordid the number of forms that were submitted. This indicates that subjects were stillentering information at the same rate in the later sessions as in the earlier ones. Sincethe proportion of requests that were visits to annotations and criteria did not increaseover the sessions, this suggests that the subjects were continuing to change their ownroles over the sessions.

The subjects did display an increased amount of interest in other people’s rolesover the sessions. This was evident both from an increased proportion ofvisits to comparisons and to the domain, and from an increased proportion ofvisits to objects outside their own role. A decrease in the proportion of visits toboth classes and individuals indicated that subjects became less concerned with thedetails of their own role (and others’), and were more interested in objects that gaveoverviews of roles.

Sequence analysis showed that standard routes were taken through APECKS by thesubjects, and a distinction between the first session and the later sessions can be seen inthe pattern of actions. During the first session subjects tended to visit the ‘‘Actions’’pages of their roles and classes. In later sessions, subjects followed the pattern of domain

main page ! own role’s main page ! list of comparisons with own role ! comparison

between own & others’ role’s main page. Both these patterns of transitions were thoseshown to the subject during the instructions. It is interesting that they stuck stubbornlywith the latter pattern in the later sessions, despite the quicker possible jump straightfrom the domain’s main page to a comparison’s main page. This may have beenbecause they wished to stick with a known sequence of actions, or because it was easierto identify comparisons between their own role and other people’s in this way.


5. APECKS andWebGrid

5.1. WEBGRIDAS EXTERNAL KA SUPPORT

The use of WebGrid-II as an external KA support provided interesting problems notonly for APECKS, but also for WebGrid, which was not originally designed for use byexternal applications in precisely the way that APECKS required. In particular, it doesnot make its output available to external users by encoding it into a standardrepresentation language, such as KIF, but instead uses a special encoding withinHTML pages. Hence, an interesting requirement for APECKS was to incorporateWebGrid output without distorting WebGrid’s operation.

When WebGrid is being accessed through APECKS, APECKS is acting as an HTTPclient towards WebGrid, while still acting as a server for the user’s WWW browser.Throughout the interaction, the user is presented with APECKS’ standard navigation bar,so that the transfer of inference and the general navigation are as clear and transparent aspossible. There are three processes that APECKS runs to achieve this effect.

5.1.1. Translating ontologies into grids. The first of these processes is translatingstandard APECKS ontologies, represented in the frame language discussed in Section3.2.2, into the WebGrid repertory grid format, as hidden fields in HTML pages. Toinitiate an interaction with WebGrid, APECKS must first generate a form submissionthat, submitted to WebGrid, will encode the representation of the role (or part of role)to be analysed. Once the translation is carried out and the field submitted, the user seesthe WebGrid main page, with prompts for triad elicitation, construct and elementediting, etc.

The translation problem, from APECKS’ hierarchical representation to the flat oneused by WebGrid, is a tricky one. Although there is the functionality within WebGridfor exposing hierarchical relationships within grids (Shaw & Gaines, 1998) using ratingson constructs, these dimensions cannot be supplied by users initially. Furthermore,recall that APECKS is intended to be usable by domain experts without knowledgeengineering experience, and the ability to interpret the outcome of grid analyses cannotbe assumed in the user. The graphical output would be more intuitive, but is notmachine interpretable.

The APECKS solution is to isolate three separate grids within the relevant role, andthen to offer users the option of analysing any of these grids at a time. The simplesttranslation represents each APECKS instance as a WebGrid element, and each slot andclass represented as a construct. Then, for example, the subclass partition ‘‘colour’’ inFigure 5 above, with its two classes ‘‘light’’ and ‘‘dark’’, would be translated into acategorical construct taking the values ‘‘light’’ and ‘‘dark’’. The slot ‘‘hardness’’ wouldbe represented as a rating scale construct and the class ‘‘acid rocks’’ would be a Booleanscale construct, taking the values 1 (‘‘in-acid-rocks’’) or 2 (‘‘not-in-acid-rocks’’). Whereclass membership is not specified, a ‘‘?’’ is given as the value.

The second type of grid takes APECKS classes as elements in WebGrid, with theconstructs based on the slots, and the values of the constructs being the default valuesfor the instances of the class. For example, all the subclasses of the class ‘‘rocks’’, suchas ‘‘large grainsize’’, ‘‘small grainsize’’ and ‘‘medium grainsize’’ would be represented aselements. The slot ‘‘grainsize’’ would be represented as a categorical construct, taking


the value ‘‘large’’ for the element ‘‘large grainsize’’, ‘‘medium’’ for ‘‘medium grainsize’’and so on. This translation allows users to analyse the class hierarchy.

The third type of grid allows users to gain an overview of the types of slot availablewithin a role. This enables them to discover or set such properties as inverses,inheritedness, symmetricality, etc. The elements are the various internal properties ofslots, the constructs are the slots defined and the values are the slots’ values on theproperties defined. Hence, for example, a slot ‘‘siblings’’ could be shown by thisencoding to be its own inverse, symmetric, irreflexive and weakly transitive (i.e. if aRband bRc, aRc iff aac), while ‘‘ancestor’’ would be the inverse of ‘‘descendant’’,asymmetric, irreflexive and transitive. In the Sisyphus example, the slot ‘‘minerals’’,which represents the minerals that are present in a type of rock, and the slot ‘‘rocks’’,which represents the rocks that types of mineral can be part of, would be represented aselements in WebGrid. The construct ‘‘inverse’’ would hold the value ‘‘rocks’’ for theelement ‘‘minerals’’ and the value ‘‘minerals’’ for the element ‘‘rocks’’.

HTML hidden fields are constructed automatically which encode these grids fromthe APECKS pages that prompt for actions. When users submit these forms, theinteraction with WebGrid begins and APECKS acts as a gateway, the second processrequired for APECKS to interact with WebGrid.

5.1.2. APECKS as a WWW Gateway. APECKS acts as a WWW gateway while theuser is interacting with WebGrid, with all the HTTP requests to and responses fromWebGrid passing through it. Requests pass from the user’s browser untouched, butWebGrid’s responses go through three transformative processes.

(1) The WebGrid fields contained within the response are logged.(2) URLs referring directly to WebGrid are altered so that the transaction continues to

use APECKS as a gateway.(3) The most important and complex check to be made is for changes that, though

valid in WebGrid, would cause problems for APECKS. The fields from theprevious submission are checked to ensure that the user has not renamed elements,changed class membership encoding, or made other alterations that would hamperconsistent translation to and from APECKS and WebGrid. If such a change hasbeen made, the user is warned and the change undone.

5.1.3. Translating grids into ontologies. The final process is the inverse translation fromgrids to ontologies. When a user requests a page within a role that may have beenaltered on the basis of a WebGrid interaction, the recorded fields from the most recentresponse from WebGrid are used to make changes to the APECKS ontology. Changesto APECKS are postponed until then to reduce the amount of processing requiredwhile the user is interacting with WebGrid. The user, of course, does not need to signalthe end of his use of WebGrid, as this would compromise the aim of a transparentinteraction between it and APECKS.

5.2. WEBGRIDAS EXTERNAL COMPARISON SUPPORT

We have seen in Section 4.1.3 how APECKS deals with comparisons, and also thatWebGrid can be used within APECKS for comparison support in Section 4.4. This is


achieved by a conversion of APECKS roles to WebGrid grids as described inSection 5.1.1.

The result of these comparisons is a number of pages that indicate to users howdifferent roles compare, and give prompts for the users to investigate differences whereappropriate (the user always being free to disregard a prompt, of course). Currently, inthe absence of a comparison engine that gives programmatic analysis of itscomparisons, users are only given prompts concerning the class hierarchies in thecompared roles. They may then change instances or the name of a class in one of theirroles, or may choose to retain the current structure and annotate it to make the designrationale clear.

The results, as we have seen in Section 4.4, are generally informative and interesting.(Tennison & Shadbolt, 1998) discusses the ways that such grids represent comparisonsof ontologies in some detail. However, one problem is particularly relevant here, andthat is that a number of rows on the grid can end up giving information that is not veryuseful. The technique can throw up correspondences that are merely artefacts of theways that multiply valued slots are encoded into WebGrid constructs, and of thecomparison techniques themselves.

For example, in the Sisyphus III domain discussed in Sections 4.2 and 4.3, it might beuseful to include a slot within the ontology stating which minerals are present in eachrock. For the purposes of comparison, such a multi-valued slot would be translated intomultiple Boolean rating scale constructs, one for each possible mineral, indicatingwhether or not each mineral was present in the rock. This can lead to constructs thathold the same value across all elements.

6. Discussion

APECKS has been described from a number of angles, showing the principles behind it,the way it is used, and its relation with WebGrid. We will complete our account bydiscussing some of the issues that have been raised by the development and testing ofAPECKS.

6.1. THE NATURE OF ONTOLOGIES

A discussion underlying the development of APECKS has only been hinted at in thispaper, though it was addressed directly in Tennison and Shadbolt (1998), and that is anargument over the nature of ontologies. In Section 2.2, we introduced two strategies forontology maintenance, the centralized model, where one person takes responsibility forthe content of the ontology or part of it, and a distributed style of management wherepeople add content when they think it is appropriate. These strategies}which of coursecan be modified, as in Smith and Farquhar (2000) which modifies the centralizedstrategy, or as in the personal roles idea from APECKS (Section 3.1.1) which modifiesthe distributed strategy}are each part of a wider mindset about how an ontology isconceived.

For many, an ontology is best seen as a piece of work that, in some way, expresses anobjective account of the objects and concepts in a domain, and then when a particularKB is required for a task or application, such an objective account can be raided, using


techniques such as those suggested by Swartout et al. (1996). Alternatively, smaller-scale ontologies can be brought together to create bigger, more widely applicable ones.Whichever method is used, knowledge engineers building ontologies remain concernedabout achieving ideals of consistency and correctness. And, as we have argued in Section2.3, it may be that an early collaborative stage of ontology development is a usefuladdition to the methodology of constructing such ontologies. In that event, a serverthat supports collaborative construction such as APECKS would prove invaluable insuch an environment.

However, there is an alternative view, discussed in Tennison and Shadbolt(1998). The WWW has facilitated the animation and expansion of documents; suchliving documents can evolve and adapt on the basis of the needs of the user.Analogously, Tennison and Shadbolt (1998) argued that living ontologies should bedeveloped. Such ontologies would evolve both in response to domain evolution, and asa result of collaboration between contributors. They could play the same role asstandard ontologies within knowledge engineering, or become a medium for thedevelopment of a consensual position (or alternatively for the statement of argument orcontroversy).

Furthermore, such a commitment to develop living ontologies can cut the fixed, orsunk, costs of knowledge codification in domains where the knowledge itself iscontested (cf. Cowan & Foray, 1997, and Section 2.3). The development of a consensualposition}or, rather, the prerequisite for this, which is the development of anuncontested language for describing the domain}is an expensive business which,before it is done, makes the cost/benefit analysis of ontology construction relativelyunfavourable.

Such a philosophy puts a series of constraints upon a living-ontology server. Inparticular, it must support the development of competing ontologies in parallel. It mustalso support the comparison of such ontologies, and any discussion that follows, forotherwise it would not be properly supporting the achievement of consensus.

6.2. KNOWLEDGE ENGINEERING SANS KNOWLEDGE ENGINEERS

Making such a server available over the web means that contributors would not beconfined to particular research groups at particular moments, but dispersed in spaceand time. Consensus achieved should be seen as temporary and liable to be overturnedin future. A server must therefore support direct use by domain experts; it would be anunacceptable restriction on the breadth and flexibility of debate if a knowledge engineerhad to be present at every interaction. An aim of living ontologies is to empower expertsto add to and enrich debate; if they came with a requirement that a knowledge engineeralways acts as mediator, then only a relatively small number of experts couldparticipate. This is not to say that knowledge engineers can or should have no role insystem development, only that the overhead they impose on ontology construction can,in some circumstances, act as a barrier to entry.

However, dropping the knowledge engineer out of the process also has the side effectof removing an honest broker from the process of reaching consensus. This raises thespectre of a living ontology being a free-for-all pursuit of narrow professional interestsby each contributor, to the exclusion of meaningful collaboration.


Without the use of force it is, of course, impossible to ensure that experts willcollaborate in a spirit of scientific disinterest. But one way of promoting consensus is tosupport the examination of the causes of divergence. Making what may be implicitdifferences between experts explicit may go some way towards defusing difficulties, andtherefore enhancing the understanding of the domain. And of course, it is generally thecase that most users of a system genuinely attempt to make it work, and very fewinterested users actively attempt to subvert such initiatives. If domain experts haveliterally no interest in promoting consensus, then APECKS cannot help them; this, ofcourse, is true of any ontology server, whether it is intended to provide livingontologies, or the more usual moribund ones.

APECKS aims to enable groups of experts to address the sources of theirdisagreements, and to promote consensus where possible. Again, the fixed costs ofany such initiatives are large, and removing knowledge engineers from the process canonly help bring those costs down. This is, of course, also generally consonant withrecent speculations about the future of knowledge engineering and knowledgemanagement (e.g. Smith & Farquhar, 2000).

A further issue which needs to be addressed is that of how best to representontologies and other knowledge engineering concepts to users in the absence ofknowledge engineers. For instance, virtual realities may be employed (e.g. to show anontology of cars using an image of a car), or to exploit the expressive possibilities ofvisual metaphor (e.g. representing a hierarchy as a tree). Such developments dependrather on the results of evaluating systems like APECKS, and it is to the evaluationeffort we now turn.

6.3. EVALUATION

6.3.1. Evaluation of KA tools and techniques: general problems. The evaluation ofknowledge acquisition tools has been a serious problem (Shadbolt et al., 1999). Thenotion of rigorous evaluation carries with it a number of difficulties, which wereitemized by Shadbolt et al. as follows.

* Because KA software is designed to elicit quality knowledge from domain experts, thereare problems in assembling statistically significant samples: domain experts are scarce.

* It cannot be assumed that domain experts are homogeneous, as experimentalsubjects ought to be.

* The quality and completeness of the knowledge acquired is hard to confirm.* The value of KA tools varies dramatically across contexts (e.g. task types).* A negative evaluation may be due to the tool’s underlying methodology, or merely

for that particular implementation of the methodology.* There are no unproblematic metrics.

These are particular problems with respect to the evaluation of tools; evaluatingmethodologies or other types of approach are even more fraught with pitfalls. See(Shadbolt et al., 1999) for an in-depth review of the difficulties in evaluating KA tools,models and methodologies.

The conclusion of Shadbolt et al. was that experimentation was the best, if neither aperfect nor the only, solution for the evaluation problem [not, as Groot, van Harmelen


& ten Teije (2000) suggest was concluded, that empirical, qualitative evaluation is theonly possibility]. The value of experimentation increases dramatically as the number ofexperiments performed throughout the field increases. At present, though, fewevaluation exercises (experimental or otherwise) have been carried out within KA.Partly this is a resource problem, for it is much easier to garner money for buildingprototypes than for evaluating them; partly it is a cultural problem within the field [seeShadbolt et al. (1999) for a discussion of the unsatisfactory response to the Sisyphus IIIexperiment].

We should also point out that, following the completion of the evaluation describedin this paper, WonderTools, a useful framework for evaluating ontological engineeringtools has been reported (Duineveld et al., 2000). This framework approaches evaluationfrom a different direction than our own work, in that it is designed for use byprofessionals rather than users. This does mean that evaluations using the frameworkwould not necessarily uncover the serious difficulties in usability, although it doesprovide a thoroughgoing programme for independent evaluation which is of course nobad thing. Furthermore, the resource requirement for an evaluation along these lineswould probably have been in excess of what was available for APECKS. Nevertheless,this is an important contribution to the process of evaluating tools for ontologycreation which would have been influential on our work had it been available at thetime of the experiment.

6.3.2. Lessons from the evaluation of APECKS. APECKS is a system with manyfeatures. Practical considerations}the genesis of the system was in a Ph.D. thesis(Tennison, 1999)}limited the extent of the evaluation and its generality. These arediscussed here as an explanation for some of the decisions made in the design of theevaluation study.

(1) There are few, if any, evaluations of other ontology servers that could serve as abaseline against which APECKS could be evaluated. Equally, it is difficult to carryout detailed evaluations of other systems as logging and automated quantitativeevaluations of ontologies are not generally accessible. Without such comparisonsto other systems, it cannot be said whether APECKS is better or worse than otherontology servers.

(2) A difficulty was finding people who would be willing to use the system within thebudgetary constraints of the evaluation study (d10 per subject!!). There is anobvious cost in using professionals for an evaluation of a proof-of-concept system;equally it was important that the subjects display some expertise. For this reason,undergraduate students were used. As argued in Shadbolt et al. (1999), this shouldbe no impediment to the acceptance of the evaluation results.

(3) There were multiple constraints on the choice of knowledge domain. The domainhad to be one in which undergraduate students could be reasonably considered‘‘expert’’, and in which there was enough structure to allow meaningful debate totake place. In the end, the domain of ‘‘mammals’’ was selected, as having theseproperties.

(4) APECKS is designed for extended use, but there was a limit to the amount of timesubjects could spend using the system. APECKS is also designed for people using itat the same time as others, as well as before and afterwards. However, in order to


monitor their use of the system, subjects had to have separate sessions, and in orderfor them to achieve anything during these sessions, they had to be of a reasonablelength. These issues were resolved by having four sessions per subject, distributedover a 3-week period.

Overall, the evaluation showed that, with a few provisos, APECKS was a usablesystem. The more the subjects used APECKS, the easier they found it to use. They wereable to produce knowledge representations using APECKS alone, and used the KAsupport built into APECKS (including WebGrid-II) in order to produce theirknowledge representations.

From their comments, they found the comparisons between their own and others’roles to be very useful, and used them to improve their own role, to investigate others’roles, and to make comments on others’ roles. The subjects became less egocentric andmore interested in other people’s roles over the course of the experiment.

These findings are particularly positive considering the hurdles that the subjects hadto overcome. They were given a relatively short amount of time to construct theirontologies, considering that APECKS is designed for extended use. The subjects had abiological background, but were not specialists either on mammals or on taxonomicclassification. They had no previous experience in knowledge engineering.

However, the comments provided by the subjects show that the presentation ofinformation, the navigation system and the discussion support all need to be improved.Much of these specific criticisms could be addressed by an overhaul of the user interface(e.g. including a ‘‘back’’ button, and button to return the user to the main page of hisown role would help with most reported navigation problems).

These generally positive conclusions must of course be tempered withan understanding of their limitations. But the data collected during thisevaluation can serve as a valuable baseline for future comparisons with, for examplethe following.

* Other ontology servers, to see if APECKS is an improvement over those that alreadyexist.

* Other ontology servers used in conjunction with a collaborative virtual environment,to discover whether APECKS’ integrated discussion system provides more thansimilar functionality from existing systems.

* APECKS without comparison support, to investigate whether automated compar-isons help users to discuss and compare ontologies.

* Use by knowledge engineers, to evaluate their experience compared to domainexperts.

* Use in different domains and with different tasks, to ascertain how use changes withdifferent domains and tasks.

* Use over an extended time frame, to study the effects of long-term use.

There are inherent problems with evaluating KA software. These are partly to dowith the difficulties of assembling enough subjects capable of using the software toprovide significant results, partly with the difficulties of evaluating the quality of anyknowledge acquired, and partly with the difficulties of separating out the respectivecontributions made to the results by methodology, individual methods and


implementation details (Shadbolt et al., 1999). Clearly, these problems cannot alwaysbe overcome. However, the exercise of evaluating a methodology-driven piece ofsoftware as reported in Section 4.5 has shown that it is possible to surmount at leastsome of these difficulties some of the time, and to secure interesting and significantresults that point the way towards improvements in prototype systems. Although theeconomics of the KA field tend to dictate that system building attracts more fundingthan system evaluation, relatively cheap evaluation experiments can provide invaluablefeedback for the production of KA tools that meet the needs of their users, not theircreators.

6.4. WEB-ACCESSIBLE TOOLS AS EXTERNAL SUPPORT

A very encouraging result of the development of APECKS is proving the concept ofproviding an environment where web-accessible tools can be incorporated seamlesslyinto users’ ontology development efforts. There was an effort involved in the use ofWebGrid, largely to do with the difficulties of translating between APECKS’hierarchical structure, and WebGrid’s flat representations (Section 5).

However, WebGrid’s history and associations with repertory grid analysis haveresulted in what is, for ontological purposes at least, a relatively unusual type ofrepresentation. Most ontologies are currently written in something like the frame-basedlanguage of APECKS itself, thanks largely to various standardization efforts of whichOntolingua is the most prominent (Farquhar et al., 1997).

Hence, it is not impossible to imagine an APECKS-like tool providing anenvironment where various KA and ontology construction tools are brought downfrom the web, with minimal translation effort, for use by a variety of people dispersedin space, time and research community. Furthermore, other ontology servers could beraided for seed ontologies, or other ontologies for comparison, comment or use as goldstandard.

Nevertheless, in the context of the semantic web, and its tremendous potential as astore of information, knowledge and computational resources, the early results fromAPECKS are highly encouraging. It supports a vision of a software environment thatcan direct non-expert users towards the software that is likely to be useful for solvingtheir particular problems, and this idea of the provision of knowledge services is surelyone that can be explored in more detail as the representational potential of the semanticweb begins to be exploited in full.

6.5. APECKS AND THE FUTURE

As noted above, APECKS is an experimental prototype that was designed as a proof ofconcept, rather than a commercially or academically exploitable tool. The currentmarket leader in ontology development environments is Prot!eeg!ee-2000, which shareswith APECKS a number of principles of flexibility and adaptability (Noy et al., 2001),with a usable graphical and customizable user interface that keeps the developer atarm’s length from the internal representations. Furthermore, being open source it canevolve at the behest of the ontological development community.

However, many of the interesting and useful features of APECKS do not appear inProt!eeg!ee. For instance, Prot!eeg!ee does not provide support for annotation, although as we


have seen, annotation is a central plank of a consensual ontology designprocess. Neither does the architecture support the collaborative and criticalinterchanges that APECKS has advocated. On the other hand, Prot!eeg!ee, of course,has impressive knowledge acquisition support that will be essential in a world whereknowledge engineering has to become routine (Smith & Farquhar, 2000). APECKS’strategy here was to ‘‘buy in’’ KA support from outside knowledge service providers(and of course this might have included support from Prot!eeg!ee itself); the advantage ofthis eclectic stance is that}given appropriate translations between representationlanguages}the APECKS user could in theory use whichever KA technique wasavailable over the web.

APECKS is a proof of concept, not only of a tool, but also of a methodology forconsensual ontology construction, criticism and maintenance. Much of its functionalitywas determined by a view of how ontologies will be developed in the future (Tennison,1999), and the evolution of the semantic web has borne out that vision (Smith &Farquhar, 2000; Berners-Lee et al., 2001). We have argued that it has many advantagesfor the development and maintenance of ontologies, and that many aspects of itsphilosophy of ontology building could fruitfully be incorporated into the nextgeneration of ontology development tools.

This paper is based primarily on work performed by Jeni Tennison in her Ph.D. thesis for theArtificial Intelligence Group, School of Psychology, University of Nottingham (Tennison, 1999).The evaluation study was carried out with the assistance of the School of Life Sciences at theUniversity of Nottingham. The writing of the paper was supported by the Advanced KnowledgeTechnologies (AKT) Interdisciplinary Research Collaboration (IRC), which is sponsored by theUK Engineering and Physical Sciences Research Council under grant number GR/N15764/01.The AKT IRC comprises the Universities of Aberdeen, Edinburgh, Sheffield, Southampton andthe Open University. The authors would also like to thank the anonymous referees for theirhelpful comments.

References

Berners-Lee, T., Hendler, J. & Lassila, O. (2001). The semantic web. Scientific American,284, 35–43.

Boy, G. (1996). The group elicitation method: an introduction. In N. Shadbolt, K. O’Hara &G. Schreiber, Eds. Advances in Knowledge Acquisition, pp. 290–305. Berlin: Springer-Verlag.

Buckingham Shum, S. & Sumner, T. (1998). New scenarios in scholarly publishing and debate.In M. Eisenstadt & T. Vincent, Eds. The Knowledge Web: Learning and Collaborating onthe Net, pp. 135–151. London: Kogan Page.

Chandarasekaran, B., Josephson, J. R. & Benjamins, V.R. (1998). The ontology of tasksand methods. Proceedings of the 11th Knowledge Acquisition Workshop (KAW’98).Banff, Canada, April 18–23, 1998. Available at http://ksi.cpsc.ucalgary.ca/KAW/KAW98/chandra.

Conklin, E. J. & Yakemovic, K. C. B. (1991). A process-oriented approach to design rationale.Human–Computer Interaction, 6, 357–391.

Corbet, G. B. & Hill, J. E. (1991). A World List of Mammalian Species (3rd edn.). OxfordUniversity Press, London: Natural History Museum Publications.

Cowan, R., David, P. A. & Foray, D. (1999). The explicit economics of knowledge codificationand tacitness. Presented at 3rd TIPIK Workshop, Strasbourg, 24th April, 1999. Available athttp://www-econ.stanford.edu/faculty/workp/swp99027.pdf.


Cowan, R. & Foray, D. (1997). The economics of codification and the diffusion of knowledge.Industrial and Corporate Change, 6, 595–622.

Curtis, P. (1992). Mudding: social phenomena in text-based virtual realities. Proceedings of the1992 Conference on the Directions and Implications of Advanced Computing, Berkeley.Available as Xerox PARC Technical Report CSL-92-4, ftp://ftp.parc.xerox.com/pub/MOO/papers/DIAC92.ps.

Dewey, J. (1929). The Quest for Certainty. New York: G.P. Putnam.Domingue, J., Motta, E., Buckingham Shum, S., Vargas-Vera, M., Kalfoglou, Y. &

Farnes, N. (2001). Supporting ontology driven document enrichment within communitiesof practice. Proceedings of K-CAP ‘01,Victoria, Canada.

Duineveld, A. J., Stoter, R., Weiden, M. R., Kenepa, B & Benjamins, V. R. (2000).WonderTools? A comparative study of ontological engineering tools. International Journalof Human-Computer Studies, 52, 1111–1133.

Eisenstadt, M. & Vincent, T., Eds. (1998). The Knowledge Web: Learning and Collaborating onthe Net. London: Kogan Page.

Euzenat, J. (1996). Corporate memory through cooperative creation of knowledge bases andhyper-documents. Proceedings of the 10th Knowledge Acquisition Workshop (KAW’96).Banff, Canada, November 9–14, 1996. Available at http://ksi.cpsc.ucalgary.ca/KAW/KAW96/euzenat/euzenat96b.html.

Farquhar, A., Fikes, R. & Rice, J. (1997). The Ontolingua server: a tool forcollaborative ontology construction. International Journal of Human–Computer Studies,46, 707–727.

Fensel, D., Angele, J., Decker, S., Erdmann, M., Schnurr, H.-P., Staab, S., Studer, R. &Witt, A. (1999). On2broker: semantic based access to information sources at theWWW. Proceedings of the World Conference on the WWW and the Internet (Webnet 99),Honolulu, Hawaii, October 25-30, 1999. Available at www.cs.vu.nl/�dieter/ftp/paper/webnet.pdf.

Fensel, D., Horrocks, I., van Harmelen, F., Decker, S., Erdmann, M. & Klein, M. (2000).OIL in a nutshell. In R. Dieng & O. Corby, Eds. Knowledge Engineering and KnowledgeManagement: Methods, Models and Tools, pp. 1–16. Berlin: Springer-Verlag.

Fensel, D., van Harmelen, F., Horrocks, I., Mcguinness, D. L. & Patel-Schneider, P.F.(2001). OIL: an ontology infrastructure for the semantic web. IEEE Intelligent Systems, 16,38–45.

Fernandez, M., G !oomez P!eerez, A., Pazos, J. & Pazos, A. (1999). Ontology of tasks andmethods. IEEE Intelligent Systems and Their Applications, 14, 37–46.

Gaines, B. R. & Shaw, M. L. G. (1996). A networked, open architecture knowledgemanagement system. Proceedings of the 10th Knowledge Acquisition Workshop (KAW’96).Banff, Canada, November 9–14, 1996. Available at http://ksi.cpsc.ucalgary.ca/KAW/KAW96/gaines/KM.html.

G !oomez P!eerez, A. & Benjamins, V. R. (1999). Overview of knowledge sharing and reusecomponents: ontologies and problem-solving methods. In V. R. Benjamins, B. Chan-drasekaran, A. G !oomez P!eerez, N. Guarino & M. Uschold, eds. Proceedings of theIJCAI-99 Workshop on Ontologies and Problem-Solving Methods, Stockholm, Sweden.

Groot, P., van Harmelen, F. & ten Teije, A. (2000). Torture tests: a quantitative analysis forthe robustness of knowledge-based systems. In R. Dieng & O. Corby, Eds. KnowledgeEngineering and Knowledge Management: Methods, Models and Tools. Berlin: Springer-Verlag.

Gruber, T. R., Tenenbaum, J. M. & Weber, J. C. (1992). Toward a knowledge medium forcollaborative product development. In J.S. Gero, ed. Artificial Intelligence in Design ’92.Proceedings of the Second International Conference in Design, pp. 413–432. Dordrecht:Kluwer Academic Publishers. Available at ftp://ksl.stanford.edu/pub/knowledge-sharing/papers/shade.ps.

Habermas, J. (1968/1987). Knowledge and Human Interests. Oxford: Polity Press.Karp, P. D. & Gruber, T. (1997). The Generic Frame Protocol. Available at http://

www.ai.sri.com/�gfp/spec/paper/paper.html.


Klein, M. & Fensel, D. (2001). Ontology versioning on the semantic web. Proceedings ofSWWS 2001, Stanford. Available at http://www.semanticweb.org/SWWS/program/full/paper56.pdf.

Lakatos, I. (1970). Falsification and the methodology of scientific research programmes. In I.Lakatos & A. Musgrave, Eds. Criticism and the Growth of Knowledge, pp. 91–196.Cambridge: Cambridge University Press.

MacClean, A., Young, R. M., Bellotti, V. M. E. & Moran, T. P. (1991). Questions, Options,and Criteria: elements of design space analysis. Human-Computer Interaction, 6, 201–250.

Maedche, A. & Staab, S. (2001). Comparing Ontologies } Similarity Measures and aComparison Study. AIFB Internal Report no.408, Karlsruhe. Available at http://ontobroker.semanticweb.org/ontos/report-aifb-408.pdf.

Major, N. P. (1992). CATO: an automated card sort tool. In M. Linster & B. Gaines, Eds.Proceedings of EKAW’91. GMD-Studien No. 211, September 1992.

McGuinness, D. L., Fikes, R., Rice, J. & Wilder, S. (2000). An environment for merging andtesting large ontologies. Proceedings of the 7th International Conference on Principles ofKnowledge Representation and Reasoning (KR2000), Breckinridge, CO.

Motta, E., Buckingham Shum, S. & Domingue, J. (2000). Ontology-driven documentenrichment: principles, tools and applications. International Journal of Human-ComputerStudies, 52, 1071–1109.

Mulholland, P., Domingue, J., Zdrahal, Z. & Hatala, M. (2000). Organizational learning:an overview of the ENRICH approach. Journal of Information Sciences and Use,20, 9–23.

Noy, N. F. & Musen, M. A. (2000). PROMPT: algorithm and tool for automated ontologymerging and alignment. Proceedings of the Seventh National Conference on ArtificialIntelligence (AAAI-2000), pp. 450–455. Austin, TX. Available at http://www-smi.stanfor-d.edu/pubs/SMI Reports/SMI-2000-0831.pdf.

Noy, N. F., Sintek, M., Decker, S., Crubezy, M., Fergerson, R.W. & Musen, M. A. (2001).Creating semantic web contents with Prot!eeg!ee-2000. IEEE Intelligent Systems, 16, 60–71.

Quine, W. V. O. (1969). Ontological relativity. In W. V. O. Quine, Ed. Ontological Relativityand Other Essays, pp. 26–68. New York: Columbia University Press.

Selvin, A., Buckingham Shum, S., Sierhuis, M., Conklin, J., Zimmermann, B., Palus, C.,Drath, W., Horth, D., Domingue, J., Motta, E. & Li, G. (2001). Compendium: makingmeetings into knowledge events. Proceedings of Knowledge Technologies 2001, Austin, TX.Available at http://kmi.open.ac.uk/tr/papers/kmi-tr-103.pdf.

Shadbolt, N., O’Hara, K. & Crow, L. (1999). The experimental evaluation of knowledgeacquisition techniques and methods: history, problems and new directions. InternationalJournal of Human–Computer Studies, 51, 729–755.

Shaw, M. L. G. & Gaines, B. R. (1989). Comparing conceptual structures: consensus, conflict,correspondence and contrast. Knowledge Acquisition, 1, 341–363.

Shaw, M. L. G. & Gaines, B. R. (1998). WebGrid-II: developing hierarchical knowledgestructures from flat grids. Proceedings of the 11th Knowledge Acquisition Workshop(KAW’98). Banff, Canada, April 18–23, 1998. Available at http://repgrid.com/reports/KBS/WG/.

Simpson, G. G. (1945). The principles of classification, and a classification of mammals. Bulletinof the American Museum of Natural History, 85, 1–350.

Smith, R. G. & Farquhar, A. (2000). The road ahead for knowledge management: an AIperspective. AI Magazine, Winter 2000, 21(4), 17–40.

Studer, R., Benjamins, V. R. & Fensel, D. (1998). Knowledge engineering: principles andmethods. Data and Knowledge Engineering, 25, 161–197.

Swartout, B., Patil, R., Knight, K. & Russ, T. (1996). Toward distributed use of large-scaleontologies. Proceedings of the 10th Knowledge Acquisition Workshop (KAW’96). Banff,Canada, November 9–14, 1996. Available at http://ksi.cpsc.ucalgary.ca/KAW/KAW96/swartout/Banff 96 final 2.html.

Tennison, J. (1999). Living Ontologies: Collaborative Knowledge Structuring on the Internet.Ph.D. Thesis, Deptartment of Psychology, University of Nottingham.


Tennison, J. & Shadbolt, N. (1998). APECKS: a tool to support living ontologies. Proceedingsof the 11th Knowledge Acquisition Workshop (KAW’98). Banff, Canada, April 18–23, 1998.Available at http://ksi.cpsc.ucalgary.ca/KAW/KAW98/tennison.

Welty, C. & Guarino, N. (2001). Supporting ontological analysis of taxonomic relationships.Data and Knowledge Engineering, 39, 51–74.

Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity. Cambridge:Cambridge University Press.

Zdrahal, Z. & Domingue, J. (1998). The world wide design lab: an environment for distributedcollaborative design. In M. Eisenstadt & T. Vincent, Eds. The Knowledge Web: Learningand Collaborating on the Net, pp. 225–238. London: Kogan Page.

APECKS: using and evaluating a tool for ontology construction with internal and external KA support

Documents

Transcript of APECKS: using and evaluating a tool for ontology construction with internal and external KA support