Semantic Multimedia

Steffen Staab

ISWeb – Informationssysteme & Semantic Web

Semantic Multimedia

Steffen Staab, Univ. Sheffield

March 28, 2006

Steffen Staab


My private challenge

….and more than 17,000 other images and mini-movies

Steffen Staab


What I would like to do… vs What I can do

• Send family recent christmas photos…

• Send friends pictures that include them…

• Ask for pictures that depict my children at carnival with a big smile in order to make a presentation about semantic multimedia…

• Show a friend where we live…• Exchange opinions about what

are the best shots • Record „photo copies“ of signed

contracts• Query for all architecture images

built for X-Media proposal…

List to be continued…

• Store pictures in a folder• Query for picture name and

date

The Semantic Gap

Steffen Staab


Strategies for Narrowing the Semantic Gap

• Image understanding– Scene classification– People recognition (that, who)– Artifact recognition

• Context understanding

• Shared Annotation

• User Feedback

Highly domain-dependent, but so far:

little domain knowledge

Google

Flickr

Little explored wrt Semantics (e.g.Santini etal.)

Steffen Staab


Simple Annotations - TagFS

Steffen Staab


Motivation• Annotation is expensive as a dedicated

process

Allow for on-the-fly annotation from all existing applications

SemanticallyEnabled App

SemanticMetadata Repository

Arbitrary Application

Arbitrary Metadata Format

Steffen Staab


Motivation

• Retrieval strategies cannot be easily anticipated by users

No fixed schema

Steffen Staab


Concept

Allow for on-the-fly annotation from all existing applications

Annotation through a filesystem interface

No fixed schema

Avoid hierarchical filesystem organisation

Steffen Staab


Use the Filesytem to Link Arbitrary Applications with a Metadata Repository




OS KernelFAT32

WebDAV

Virtual Filesystem

?

Steffen Staab


Directories represent…Queries• filesystem paths correspond to

queries• each directory translates to a

parametrized view on the metadata repository

• views can be nested

• Example

– “/tag beach/depicts

steffen“ translates into

– depicts(„steffen“, tag(„beach“, /))

– “/” represents metadata repository

Metadata• Every file is associated with

tags given by names of super directories

• Allowing for linking of directories

• Multiple hierarchies

at creation time at exploration time

Steffen Staab


Annotated Information Objects

• listing a directory (= view) returns all information objects corresponding to the view

• information objects may be files, bookmarks, chapters, ...

• Class handlers implement file system operations (read, write, ...) for a class of information objects (files, bookmarks,…)

• RDFFS maps filesystem operations to operations on a RDF-repository

• Views and Class Handlers provide tagging for files

Steffen Staab


TagFS

Architecture II




OS KernelFAT32

WebDAV

Virtual Filesystem

Fuse

RDFFs

ViewsClassHandlers


Steffen Staab


Usage Examples: Annotate

– Take Picture of Steffen and Simon in Sheffield and store it at

/tag image/tag sheffield/depicts simon/picture1– Take Picture of Steffen in Rome and store it at

/tag rome/depicts steffen/tag image/picture2– Record video of Sheffield and store it at

/tag video/tag sheffield/video1– notice that picture1 also depicts Steffen. link picture1 to

/depicts Steffen

Steffen Staab


Usage Examples: Retrieve

–all documents related to Sheffield: /tag sheffield

picture1

video1

–all images depicting Steffen: /tag image/depicts steffen

picture1

picture2

Steffen Staab


Shared Annotations:SEA – Semantic Exchange

Architecture

Steffen Staab


Use Case: Virtual Organizations• Project members need to share, i.e. distribute and retrieve,

confidential information among each other • Different members have different roles, e.g. manager, researcher, that

require different views onto the shared dataX

-Med

ia

ContractsDeliverables

X-M

edia

Work Package 1 Work Package 2

Lucy@Sheffield Sergej@Koblenz

Steffen Staab


Use Case: Image Sharing• User X has many images

– X wants to share some images publicly, some only with dedicated persons, and some not at all

– Due to the amount of images, uploading many images to a central repository is not an option

Steffen Staab


Use Case: Image Sharing

• Neither X nor X's friends want to pay for a dedicated server or hand over their images to a server managed by a 3rd party

• They would like to user their own storage

Steffen Staab


SEA

Purpose:• Decentralized

information sharing, e.g. image sharing

• Tagging as means for– Personal and collaborative

organization of information– Information retrieval– Access control

Steffen Staab


SEA: Architecture

• RDF store for meta data

• DHT implementation for efficient distribution of shared information

Steffen Staab


SEA: Features

• Autonomy for information distribution and sharing

• Flexible information organization• Simple setup and administration of sharing

environment• Privacy, data security• Ad-hoc collaboration

Steffen Staab


Centralized vs. Distributed Sharing with SEA

Conventional informationsharing characterized by centralization

SEA follows a distributed approach

Steffen Staab


Access Control in SEA

• Access control mechanisms allow to define with whom to share data– based on taggings

• e.g. everything tagged as „public“ is public• e.g. everything tagged as „forSteffen“ is accessible

for Steffen

– based on rules for access

Steffen Staab


SEA: Data Model

• Ontological meta model

Steffen Staab


Image understanding using ontologies

Steffen Staab


What is this?

Steffen Staab


Solution

Better use context and

background knowledge

Steffen Staab


Region-Based Image Labelling

1. Find semantically meaningful regions

2. Label them with concepts

3. Infer higher level annotations from initial labellings

4. Provide user-centred, semantic annotation

The overall aim is to improve the access to multimedia content.

Steffen Staab


Image Labeling: 1. Initial, region-based

Output:• Segment Classification• Hypothesis set of

possible labels for image segments

• Degree of confidence

Scene classification

Steffen Staab


Image Labeling: 1. Initial, region-based

Output of Person/Face Detection:

• Bounding boxes for detected persons/faces

• Degree of confidence

Scene classification

Steffen Staab


Multi-Tier Image Model

Segments

Label hypotheses

Confidence values

Bounding boxes

Classification of picture

….

A1

A2

B2={l1,l2}

A1 over A2

A1 overlaps B2

…

Spatial & topological information

Steffen Staab


Multimedia Reasoning

• Aim, now: – integrate available information towards – global, – consistent and – user-oriented annotation

• 3 tasks:– Consistency Checking– Region Merging– Generation of a higher-level, user-centered annotation

Current Focus

Steffen Staab


Consistency Checking

Constraint Satisfaction Problem (CSP)

• Check that label(s) of a region are consistent wrt labels of neighboring regions

• Ideally:– Leaves one correct label per

region

• More often:– more than one label remains– decision in favor of highest

confidence values

Process consists of

1. Transformation of multi-tier description into a CSP

2. Application of constraint reasoning to solve the CSP

3. Computing the “best” labeling using the confidence values

Steffen Staab


Constraint Satisfaction Problem (CSP)

Definition• Consists of set of variables and set of constraints

relating several variables• Each variable may have values from it’s domain• A constraint defines which values can be assigned to a

variable depending on the related variables• Standard methods exist to solve the CSP• Two steps:

– Consistency checking, i.e. removal of values from the domain that never satisfy the constraints

– Computation of full solutions using search algorithms(i.e. model generation)

Steffen Staab


Constraint Satisfaction Problems

Example:• Variables:

x, y, z

• Domains:D(x) = {1, 2, 3}, D(y) = {2, 3, 4}, D(z) = {2, 3, 4, 5}

• Constraints:x >= y, y >= z

• After consistency checking: D(x) = {2, 3}, D(y) ={2, 3}, D(z) = {2, 3}

• Concrete Solutions (models):(2, 2, 2), (3, 2, 2), …

– Not all possible combinations of domain values are a solution!

Steffen Staab


Image Labelling as a CSP

• Each segment s is transformed into a variable vs

• Initial Labellings Ls are the domains of the segment variables, D(vs)=Ls

• For each spatial relation type, a constraint sp-rel(v,w) is defined– the spatial constraints define which value

combinations are legal for the given relation• e.g. left-of(v,w):={(sea,sea),(sky,sky)}, but not (sky,sea)

• If two segments s,t are related with a spatial relation sp-rel, a corresponding spatial constraint sp-rel(vs,vt) is instantiated

Steffen Staab


Initial image Segmentation Mask

Example

Steffen Staab


Sea Sky Sand Person

Region 1 0.05 0.03 0.07 1.00

Region 2 0.28 0.42 0.30 0.00

Region 3 0.54 0.74 0.32 0.00

Region 4 0.79 0.54 0.43 0.08

Region 5 0.00 0.80 0.03 0.09

Steffen Staab


ConfidenceValues

Sea Sky Sand Person

Region 1 0.05 0.03 0.07 1.00Region 2 0.28 0.42 0.30 0.00Region 3 0.54 0.74 0.32 0.00Region 4 0.79 0.54 0.43 0.08Region 5 0.00 0.80 0.03 0.09

Steffen Staab


SpatialRelations

ConfidenceValues

Sky can not be left of Sea

Sea Sky Sand Person


Steffen Staab


SpatialRelations

ConfidenceValues

Sea Sky Sand Person


Sea can not be above Sky

Steffen Staab


SpatialRelations

ConfidenceValues

Sea can not be above Sky

Sea Sky Sand Person


Steffen Staab


Definition of Spatial Constraints

• Spatial constraints form an integral part of the domain knowledge used for multimedia reasoning

• Currently they are explicitly defined by a domain expert• But:

– Seems not feasible for large amounts of concepts and relations– Preferably each constraint should be accompanied by a

confidence value, which can hardly be defined by an expert

• Idea:– Learn constraints from pre-annotated images– Allow for later refinement during run-time by user interaction.– Planned extension of M-OntoMat-Annotizer for this purpose– http://www.acemedia.org/aceMedia/results/software/m-ontomat-

annotizer.html

Steffen Staab


M-Ontomat-Annotizer

Steffen Staab


Current Status:

• Use of segment classification

• Very recently: integration of person/face detection module

Steffen Staab


Initial Evaluation

• An evaluation framework for region-based image labelling was defined within the aceMedia project.

• Ground Truth is defined on a grid-basis– a N x N grid is layered on top of each image– each cell is annotated with all depicted concepts

• For evaluation the segments of the segmentation, or the bounding boxes, are mapped to the respective cells.– For each concept it is counted how often

• the concept was found correctly, i.e. a correspondence between the segment label and a grid label is found

• the concept was found in general• the concept exists in the GT

• Based on these values precision and recall for each concept, and the overall process can be defined.

Steffen Staab


Evaluation Results

• Since the method is based on content analysis modules, we evaluated the improvement reached by applying the constraint reasoning to the segment classification.

• First, precision, recall and the F-Measure were computed for the segment classification

• Then, for the CSP method applied to the initial labelling

• Finally the average improvement was calculated

Steffen Staab


Evaluation Results

Concept

Precision

Recall

F

Sky 0.77 0.69 0.73

Sea 0.66 0.59 0.62

Sand 0.75 0.94 0.84

Person 0.33 0.65 0.44

Total 0.69 0.75 0.72

Segment Classification

Concept

Precision

Recall

F

Sky 0.78 0.91 0.84

Sea 073 0.53 0.62

Sand 0.85 0.97 0.9

Person 0.38 0.62 0.47

Total 0.76 0.82 0.79

Constraint Reasoning

• Set Up:– Evaluation with ~60 images– A 8x8 grid was used for the ground truth– The segmentation was set up to always produce 8 segments per image

• Results are promising, showing an 10% increase in average.• However, results in the overall performance are needed

Steffen Staab


Next Steps

• Soft Constraint Reasoning– Fuzzy Constraints to integrate the confidence values into the

reasoning– Incremental Constraints to flexibly add constraints during

reasoning– both should provide for more robust results and lead to better

reduction of the initial label sets• Incorporation of a region merging step

– Would enable an iterative process and a knowledge-based segmentation

• Derivation of a higher-level annotation– Currently a simple combination of confidence values is applied

• the maximum degree for each concept is kept, and each concept is added to the final annotation

– later also relations and additional concepts should be inferred

Steffen Staab


Conclusion1. Narrowing the Semantic Gap requires an

Integration of Multiple Techniques

2. Some of the techniques need not be very sophisticated – e.g. tagging

3. Some sophisticated techniques may not range very far – person recognition trained for my family doesn‘t

recognize Carsten

4. Different communities need to speak to each other

5. Large chances for the Semantic Web crowd!

Steffen Staab


Bernhard Schüler

Thank You!

Sergej Sizov

Thomas Franz

Multimedia

Web ServicesP2P &

Complex Systems

Simon Schenk

S. Mir

F. S. ParreirasB. Tausch

The wonderful worldof ontologies@ISWeb

Klaas DellschaftOlaf GörlitzRabeeh Ayaz

Carsten Saathoff

C. Ringelstein

Steffen Staab

Richard Arndt

Semantic Multimedia

Documents

Transcript of Semantic Multimedia