Experiments on Pattern-based Ontology Design

Eva Blomqvist (eva.blomqvist@istc.cnr.it), Aldo Gangemi, Valentina PresuttiSTLab, ISTC-CNR, Rome, Italy

K-CAP2009Redondo Beach (CA) – 2009-09-02

Outline

• Background on Content ODPs

• Problem – research questions

• Experiment setup

• Results of analysis

• Research questions revisited

• Conclusions and future work

Background - ODPs• It is difficult to construct good quality ontologies!

– Lack of guidelines– Best-practices exist but how can we communicate them?

• Classes of problems in ontology design that can be solved by applying common solutions=> Ontology design patterns (ODPs)

• Content ODPs – small ontologies with explicit documentation of design rationales, representing modeling “good practices”

• An ontology results from the composition of a set of Content ODPs, with appropriate dependencies, plus the necessary design expansion based on specific needs

Background – ODPs (cont.)

• A Content ODP is always associated with a General Use Case (GUC) expressed using Competency Questions (CQs)

• Example: InformationRealizationWhat are the physical realizations of this information object? What information objects are realized by this physical object?

Background – ODPs (cont.)• Collected at http://www.ontologydesignpatterns.org• Currently: Logical ODPs, Content ODPs, Re-engineering

ODPs, and Alignment ODPs

Problem• What are the benefits of ODPs in ontology

engineering? (Initial focus on ontology design based on textual requirements, using Content ODPs.)1. Are Content ODPs perceived as useful?2. Are the ontologies constructed using Content ODPs

better, in some modelling quality sense, than the ontologies constructed without patterns?

3. Are the tasks given to the participants solved faster when using Content ODPs?

4. How do participants use the Content ODPs provided, and what support would be beneficial?

Related Work

• No previous experiments on ODPs– Ontology evaluation

• Experiments on Software Patterns– The community realized pretty late that empirical

evidence had to be gathered– Groups of developers using patterns and not

using patterns• Studying the behaviour• Studying the resulting software

– Studying existing solutions

Experiment Setup• 3 sessions, 45 participants in total (inexperienced, mainly students

and researchers)• Controlled settings: same tools, same training (except session 2)

and instructions, same tasks, same patterns• Procedure – the participants…

1. filled out a questionnaire recording their background and previous knowledge…

2. participated in training on OWL modelling… 3. solved the first experiment task (Task 1) without having any insight

into ODPs… 4. filled out a questionnaire recording their experience and opinions of

Task 1… 5. participated in training on Content ODPs… 6. solved the second experiment task (Task 2) where a catalogue of

Content ODPs were available… 7. filled out a questionnaire recording their experience and opinions of

Task 2.

Experiment Setup (cont.)• Task 1:

Problem To develop an OWL ontology starting from a use case.

ContextThe Warner Bros recording label has decided to manage its own productions by means of an ontology-driven application. They provide the designers with documents describing scenarios that informally describe the typical objects and data to be stored in the knowledge base. From this documents one story is extracted and assigned to ontology designers.

Story for exercise 1: album productionThe “Red Hot Chili Peppers” are: Anthony Kiedis (vocals), Flea (bass, trumpet, keyboards, and vocals), John Frusciante (guitar), and Chad Smith (drums). During 2005, the band recorded the album “Stadium Arcadium”. The album contains 28 tracks and has been released in May 2006. It includes the track of the song “Hump de Bump”, which was composed in January 2004. The critic Crian Hiatt defines the album as "the most ambitious work in his twenty-three-year career".

Evaluation of Results

• Analysis of questionnaire responses• Evaluation of the constructed ontologies

(Mainly functional and usability evaluations)

– Coverage of problem: terminological coverage and task coverage

– Usability: presence of a naming convention, labels, comments, inverse relations, disjointness, level of axiomatization

– Modelling mistakes: incomplete solutions, solutions with shortcomings

– Pattern usage: implicit and explicit

Ontology Evaluation

• Terminological Coverage

• Task Coverage

Ontology Evaluation (cont.)

• Usability

Ontology Evaluation (cont.)

• Usability

Mistakes and Patterns• Modelling mistakes

– Task 1• Persons and roles• Information and its realization• N-ary relations

– Task 2• N-ary relations (fewer than in Task 1)

• Patterns– Task 1 – successful solutions similar to patterns– Task 2 – problems with “Situation” (n-ary relation

vocabulary)

Methodological Considerations

• Maturation– Uncertainty is introduced by the learning of subjects

during the experiment– Future experiments will use parallel groups and

switching of tasks• Choice of tasks and domain

– Tasks were designed with existing coverage of pattern catalogue in mind

– Common-sense domains – what about more “advanced” domains?

• Diversity of subjects– Expert ontology engineers are not covered

Are Content ODPs useful?

• Subjective opinions– 2/3 of participants perceive them as useful or

very useful– Only 11% (5 persons) feel they were not

useful– Other conclusions

• Participants feel that they construct better quality ontologies and that they are “guided” by the patterns

• Training is needed!

Increased Quality?

• Main improvements– Task coverage– Usability– Fewer common modelling mistakes

• What about maturation?– Reduced by varying the tasks and limited time

– but introduces uncertainty

Faster Development?

• No – rather slower at this stage

• However…– Perceived as reducing routine work– Increased discussion activity

• Pointing at new aspects?• Introducing language?

– Learning curve and teaching aspects

Pattern Support Needed?

• Finding – Matching – Selecting – Especially when number of patterns increase– Relations between patterns

• Specialization in a structured manner– According to best-practices (naming etc.)

• No specific tool support available– Upcoming: XD plugin for NeOn toolkit

Future Work

• Other domains and tasks + switching

• New target groups – experts

• Method and tool evaluations

• Other types of patterns– Presentation and representation issues

• Analysing future ontologies (when patterns are more wide-spread)

Conclusions

• Content ODPs are perceived as useful• They increase

– Perceived quality– Objective quality

• Development is not faster– …at least not initially

• Better support is needed for– Finding and selecting patterns– Structured ways of reusing them