The Generalized Ontology Evaluation Framework & the I-Choose Use case

Nicolau dePaula James Michaelis

Djoko Sigit Sayogo Grace Begany

Joanne S. Luciano Jana Hrdinova

Ontology Summit Hackathon – April 6th, 2013

Outline

• GOEF- Evaluation Methodology• I-Choose & Sustainability

Certification• I-Choose: Use Case & Ontology

Evaluating ontologies

• Ontologies are composed of intrinsic and extrinsic qualities.

• Intrinsic qualities can be decontextualized and evaluated for syntactical accuracy.

Evaluating ontologies

• But how to evaluate ontologies according to its particular context, or use case that evaluates both intrinsic and extrinsic?

• We propose a: Generalized Ontology Evaluation Framework (GOEF)

GOEF ApproachThe use case contains important contextual information needed for the evaluation. This must be extracted out from the use case, i.e. and separate what is being evaluated from the evaluation metrics, then apply the evaluations.

Two stages:

o (1) Recast use case into its contextual components:(a) Functional objective(b) Standards requirements (Design) specification(c) Semantic components required to meet the above

o (2) Evaluate components using objective metricsCreate new or use existing evaluation methods at the

appropriate levels to meet 1c, 1b and 1a.

5

Three Levels of Evaluation

6

Function Level

• Represents the top level of the use case.o i.e. the function of the intended use (for

search, for integration, for certification)

• Additionally, the primary characteristics that define the classification of the domain of the ontology (organism, aircraft, assessment instrument.).

7

Standard Level

• Represents the quality or standard that has to be met by the application (e.g. for legal, interoperability, function, compliance, etc.)

• Further specifies the domain characteristics.

8

Component Level

Identifies ontology fragments needed to fulfill the function and meet the standard. (e.g. wings, engine, wheels). In complex systems, these may be composite; many “standard” components reused (nuts and bolts used in doors and wheels)).

9

Formalizing Use Cases

10

Issue1: Methodology for formalizing use cases still needed.

I-Choose Vision

“To create interoperable data networks across communities of producers, supply

chain operators, and third party certifiers, capable of producing trusted information packages to support end

consumers to make purchasing decision based on their values”

$9.58 $9.34$8.00$2.49

(All prices normalized to US$ per pound)

Rationale for I-Choose

Use-Case 1 – Ellen goes to the supermarket

Let us imagine the following scenario. A consumer, Ellen, goes to the supermarket and is looking for a particular coffee. She is concerned with the production process of the coffee and notices that some coffees have some types of certifications, such as organic or fair trade. She finds a particular coffee which is certified by Fairtrade International, a third-party certifier.

She is interested in retrieving particular information about the certification process of this certifier. She is interested in the criteria that were used in the evaluation of that coffee, and specifically if “child labor” was used in that production process. If the certifier which gave the coffee the particular certification inspected for child labor then that information would be recorded (somehow).

The goal of the I-Choose ontology is to enable a structured way to record this information, and allow easy exchange and retrieval of it.

Certification Criteria – Information to be represented in

ontology

I-Choose Ontology

GOEF – 1st stage

How to evaluate this ontology?

** GOEF **

(1) Recast components into its parts

Potential recast ofuse case components

Function: Enable retrieval of specific compliance criteria from a specific inspection process of a particular product.

Design objective: Initial system: Satisfy consensus user criteria pre-determined by survey research [we will initially use synthetic criteria]

17

Semantic components:

Compliance Criteriaa) Children Below 15b) Under 18 dangerous workc) Preventive measures ensure safety

Certification Bodya) Flo-Cert

Standard Setter Organizationa) FairTrade Internationalb) USDA Organicc) ISO 65

Producta) Coffee

GOEF – 2nd stage

** GOEF **

(2) Evaluate components using objective metrics

Identify evaluation metrics needed (consistency, coverage, etc.) some exist and can be plugged in, others may need to be developed.

Evaluation Metrics

19

Issue2: Objective metrics need also to be formalized.

Potential evaluation metrics

Correctness:•General logical/syntactical validation•Are the right terms used (compliance criteria vs. code of conduct vs. standards) •Match information provided in the ontology to general accepted definition provided by ILO (e.g. age of 15 is accepted standard)

Completeness:•All child work criteria, and necessary characteristics included•Ability of ontology to distinguish compliant vs. non-compliant criteria

Utility:•Consumer/Consumer Advocate Questions Satisfied

20

Use Case Management in GOEF

Add a New Use Case

Or, Select a Pre Registered Use Case

About “Child Labor Compliance Criteria 3.3.7

An ontology representing the FLO-CERT compliance criteria.

Flo-Cert

Small Farmers Organization (SPO)

Components [Details]

Function [Details]

Standard [Details]

GOEF & SADI Service

Issue3:

Could (or how would) GOEF be incorporated into SADI Service

OR

Could GOEF be more formalized by using the approach of SADI Service?

BACK-UP slides from here on…

Description of I-Choose:

• I-Choose is transnational project in CTG funded by NSF Interop program and CONACYT in Mexico

• To build data architecture to support ethical consumption

• Focus on sustainable coffee products produce in Mexico and consume and distribute in Canada and in the US.

• Team of researchers with mixed skills and expertise

• Components of I-Choose System:o A set of data standards to share information across

sustainable supply-chaino A governance system

Is the information in the packaging

trustworthy?

How do I know this is not a

“Green Washing”?

What are the criteria?

Which certification

should I trust?

• There are currently 432 eco-labels worldwide

• The label available in 246 countries

o(http://www.ecolabelindex.com/)

Motivating Example: i-ChooseUse Case

26

Envision Application of I-Choose

Overview of Sustainability Certification Schemes

A certification system generally consists of a standard setter and a certification body.

o E.g. Fairtrade International (FLO) : creates standards and manages the labeling initiative

o E.g. Flo-cert : internal certification body but independent to FLO; interprets the standard into verifiable control points called compliance criteria

Certification Mark/Label

• An applicant wishing to receive the Fairtrade (FLO) certification will be evaluated against a list of compliance criteria by an inspector/auditor appointed by Flo-cert

• The result from the audit/inspection will be used for the ultimate certification decision which, in this case, will grant the particular Fairtrade International certification mark/label to be attached to the product

Figure 1. General Overview of Certification System. Ontology work to

be focused on data obtained in compliance evaluation

Overview ofCertification System

31

What is an elephant, exactly?

Existing Semantic Web Design Methodology

Data Provenance

AUDITFLO-ID YEAR AUDIT NO. AUDIT_DATE AUDIT_TYPE SERVICE AUDITOR CERT_OFFICER

24525 2010 AO-00158 30/03/2010 Initial Audit Producer-HL-Single Plantation Nyagoy-Kenya Janssen Martina

AUDIT DETAILSAUDIT NO. CC NO. COMPLIANCE CRITERIA NC_AUDIT STATUS_NC REASON_FOR_CHANGE NC_CERT CM_AUDIT

AO-00158 1.2.1.2.1 More than 50% of the members of the organization are small producerless than 50%1.2.1.2.2 Small producers are understood as: those that are not structurally dependent on permanent hired labour..etc.3.3.3.01 children below 15 indication of child labor.3.3.3.02 non-suitable work conditionindication of non-suitable work condition.

CC No. : The number of the checkpoint of the checklist used by the auditorCOMPLIANCE CRITERIA : The text of the checkpoint of the checklist used by the auditorNC_AUDIT : the non-conformity as documented by the auditor during the auditSTATUS_NC : Indicator whether the text of the non-conformity was change after the audit by the certification analyst

Data Provenance

Data provenance enables exposure of relations between consumer products and their supply chains to consumer advocates.

From an encoding perspective, there are two requirements for provenance records: •Lineage for the consumer product•Semantics for individual members of the lineage

Hackathon Clinics Information

• HC-02. The General Ontology Evaluation Framework (GOEF) & the I-Choose Use Case   

primary project contact: JoanneLuciano  (jluciano@cs.rpi.edu) James Michaelis (michaelis@cs.rpi.edu) Nicolau dePaula (ndepaula@albany.edu)• Activities will be in session:   Offline Meeting:

o Sat 2013.04.06 - 11:00am – 05:00pm EDTo Location: Winslow 2nd Floor, RPI  

Open webcast segment:o Sat 2013.04.06 - 11:00am – 12:45pm EDTo Project chat-room : http://webconf.soaphub.org/conf/room/hc-02  

Country Research Team Affiliation

U.S.

Theresa PardoJana HrdinovaDjoko Sigit Sayogo

Center for Technology in Government

Giri TayiDavid AndersenDeborah AndersenNic dePaulaGrace Begany

University at Albany

Jing Zhang Clark University

Joanne LucianoRPIJames Michaelis

Holly Jarman University of Michigan

Andrew Whitmore Simmons College

Mexico Luis Luna-Reyes,Sergio Picazo-Vela

Universidad de las Americas Pueblas

Canada Roy Rejean Cefrio

Presenters:

Nicolau dePaula is a PhD student in Information Science at the University at Albany. His interests are in e-government, knowledge management, and information policy. James Michaelis is a research assistant in RPI's Tetherless World Constellation Lab, working under Professors Deborah McGuinness and James Hendler.  His research is centered on the design of explanation interfaces for intelligence gathering systems. Additionally, he has conducted prior work on techniques for logging processing activities within Semantic Web enabled systems.

Djoko Sigit Sayogo is PhD student at Rockefeller College of Public Administration and Policy and graduate assistant at the Center for Technology in Government, the University at Albany, SUNY. He is currently involved in the I-Choose research project at the Center for Technology in Government. His research interests include e-government, collaborative network, and data sharing.

Joanne S. Luciano - Research Associate Professor, TWC, Rensselaer Polytechnic Institute