26 Oct 2005 I2RP Meeting

Temporal Aspects of Intelligent Multimedia

Retrieval and Presentation

Zhisheng Huang, Jacco van Ossenbruggen, and Lynda Hardman

CWI

The Netherlands

26 Oct 2005 I2RP Meeting

Outline

• Motivation and Background

• Temporal approaches for intelligent multimedia retrieval and presentation

• Temporal reasoning in the Semantic Web

• Research Plan

• Past work (by Zhisheng)

• Demo

26 Oct 2005 I2RP Meeting

Time and Multimedia Documents and

Presentation• Time is one of the most crucial aspects of

multimedia documents and presentation.• Audio, video, animation clips are presented with

time goes,• Image archivies are usually displayed with

timelines,• Text information describes events, time

instants, intervals/periods,• Presentation involves a series of actions

which occur in time.

26 Oct 2005 I2RP Meeting

Temporal Aspects and Their Description

• Time instant: year, date, time clock• Time intervals and events: golden age,

Monet’s born,• Temporal Modifiers: after, before, in,

during, • Temporal operators: always in the past,

sometimes in the past, never in the past, in the future, now, at present

• Action operators: doing, acting, …

26 Oct 2005 I2RP Meeting

Semantics of Time

Three Main Approaches:1. The Quantitative Time Approach: time

is a collection of time instants and intervals

2. The Temporal Logic Approach: time fluent is considered as state transitions

3. The Dynamic Logic Approach: time fluent is considered as results of taking actions

26 Oct 2005 I2RP Meeting

Temporal Retrieval vs. Temporal Reasoning

• Temporal retrieval: find entities which satisfy a temporal specification.

• Temporal reasoning: check if a temporal property holds.

• Temporal retrieval can be achieved by converting answers with “yes/no” into answers with entities.

26 Oct 2005 I2RP Meeting

Quantitative Time

• Time interval: Show all work which involve Monet in years 1850-1890;

• Time instant: Show all Dutch impressionist work in 1926;

• Time instant with time modifier: Show all Dutch impressionist work after 1926;

• Event: Show all Dutch impressionist work in year Monet died;

• Event with time modifier: Show all Dutch impressionist work after Monet died;

26 Oct 2005 I2RP Meeting

Temporal Logics

• Linear time vs. branching time• Temporal operators:

• always-in-the-past, • sometimes-in-the-past, • never-in-the-past, • at-present, • always-in-the-future,• Until• Since

26 Oct 2005 I2RP Meeting

Time in Temporal Logics

• Temporal retrieval: show the first Dutch impressionist work (in this art archive). ( Defined in terms of never-in-the-past operator and the at-present operator)

• Temporal reasoning: Is it never the case that a Cubist work occurred earlier than the first impressionist work ?

26 Oct 2005 I2RP Meeting

Dynamic Logics

• State vs. Action• A state corresponds with a time instant• An action is a set of state pairs which

correspond with the starting state and the ending state of the action.

• Dynamic operators: • [a]p: after doing action a, the property p

always holds;• <a>p: after doing action a, the property

p may holds;

26 Oct 2005 I2RP Meeting

Dynamic Actions

• Dynamic action operators: • sequential actions, • parallel actions, • action choice, • state checking• conditional action

26 Oct 2005 I2RP Meeting

Time in Dynamic Logics

• Temporal Presentation: a sequential execution of actions

• Temporal retrieval: a search action with temporal specification

• Temporal reasoning: model checking with dynamic actions

26 Oct 2005 I2RP Meeting

Temporal Reasoning in the Semantic Web

• Temporal reasoning with RDF/RDFS data

• Temporal Ontologies• Time in OWL

26 Oct 2005 I2RP Meeting

Research Plan

• Compare the three temporal approaches• Investigate their expressibility on intelligent

multimedia retrieval and presentation• Quantitative time reasoning for image

retrieval and presentation• Temporal retrieval and reasoning with the

support of the Semantic Web technology.

26 Oct 2005 I2RP Meeting

Some Past Work (by Zhisheng)

• Temporal reasoning for multi-version ontologies (2005)

• PAMELA: an intelligent presentation embodied agent (2004)

• Facial expressions for embodied agents in STEP (2004)

• XSTEP: a markup Language for embodied agents (2003)

• STEP: a scripting language for embodied agents (2002)

• WASP: Web Agent Support Program (2001)

26 Oct 2005 I2RP Meeting

Pamela: Demo