Post on 16-Jan-2016
description
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Semantic Multimedia
Steffen Staab, Univ. Sheffield
March 28, 2006
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
My private challenge
….and more than 17,000 other images and mini-movies
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
What I would like to do… vs What I can do
• Send family recent christmas photos…
• Send friends pictures that include them…
• Ask for pictures that depict my children at carnival with a big smile in order to make a presentation about semantic multimedia…
• Show a friend where we live…• Exchange opinions about what
are the best shots • Record „photo copies“ of signed
contracts• Query for all architecture images
built for X-Media proposal…
List to be continued…
• Store pictures in a folder• Query for picture name and
date
The Semantic Gap
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Strategies for Narrowing the Semantic Gap
• Image understanding– Scene classification– People recognition (that, who)– Artifact recognition
• Context understanding
• Shared Annotation
• User Feedback
Highly domain-dependent, but so far:
little domain knowledge
Flickr
Little explored wrt Semantics (e.g.Santini etal.)
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Simple Annotations - TagFS
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Motivation• Annotation is expensive as a dedicated
process
Allow for on-the-fly annotation from all existing applications
SemanticallyEnabled App
SemanticMetadata Repository
Arbitrary Application
Arbitrary Metadata Format
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Motivation
• Retrieval strategies cannot be easily anticipated by users
No fixed schema
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Concept
Allow for on-the-fly annotation from all existing applications
Annotation through a filesystem interface
No fixed schema
Avoid hierarchical filesystem organisation
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Use the Filesytem to Link Arbitrary Applications with a Metadata Repository
SemanticallyEnabled App
SemanticMetadata Repository
Arbitrary Application
OS KernelFAT32
WebDAV
Virtual Filesystem
?
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Directories represent…Queries• filesystem paths correspond to
queries• each directory translates to a
parametrized view on the metadata repository
• views can be nested
• Example
– “/tag beach/depicts
steffen“ translates into
– depicts(„steffen“, tag(„beach“, /))
– “/” represents metadata repository
Metadata• Every file is associated with
tags given by names of super directories
• Allowing for linking of directories
• Multiple hierarchies
at creation time at exploration time
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Annotated Information Objects
• listing a directory (= view) returns all information objects corresponding to the view
• information objects may be files, bookmarks, chapters, ...
• Class handlers implement file system operations (read, write, ...) for a class of information objects (files, bookmarks,…)
• RDFFS maps filesystem operations to operations on a RDF-repository
• Views and Class Handlers provide tagging for files
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
TagFS
Architecture II
SemanticallyEnabled App
SemanticMetadata Repository
Arbitrary Application
OS KernelFAT32
WebDAV
Virtual Filesystem
Fuse
RDFFs
ViewsClassHandlers
Arbitrary Application
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Usage Examples: Annotate
– Take Picture of Steffen and Simon in Sheffield and store it at
/tag image/tag sheffield/depicts simon/picture1– Take Picture of Steffen in Rome and store it at
/tag rome/depicts steffen/tag image/picture2– Record video of Sheffield and store it at
/tag video/tag sheffield/video1– notice that picture1 also depicts Steffen. link picture1 to
/depicts Steffen
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Usage Examples: Retrieve
–all documents related to Sheffield: /tag sheffield
picture1
video1
–all images depicting Steffen: /tag image/depicts steffen
picture1
picture2
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Shared Annotations:SEA – Semantic Exchange
Architecture
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Use Case: Virtual Organizations• Project members need to share, i.e. distribute and retrieve,
confidential information among each other • Different members have different roles, e.g. manager, researcher, that
require different views onto the shared dataX
-Med
ia
ContractsDeliverables
X-M
edia
Work Package 1 Work Package 2
Lucy@Sheffield Sergej@Koblenz
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Use Case: Image Sharing• User X has many images
– X wants to share some images publicly, some only with dedicated persons, and some not at all
– Due to the amount of images, uploading many images to a central repository is not an option
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Use Case: Image Sharing
• Neither X nor X's friends want to pay for a dedicated server or hand over their images to a server managed by a 3rd party
• They would like to user their own storage
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SEA
Purpose:• Decentralized
information sharing, e.g. image sharing
• Tagging as means for– Personal and collaborative
organization of information– Information retrieval– Access control
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SEA: Architecture
• RDF store for meta data
• DHT implementation for efficient distribution of shared information
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SEA: Features
• Autonomy for information distribution and sharing
• Flexible information organization• Simple setup and administration of sharing
environment• Privacy, data security• Ad-hoc collaboration
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Centralized vs. Distributed Sharing with SEA
Conventional informationsharing characterized by centralization
SEA follows a distributed approach
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Access Control in SEA
• Access control mechanisms allow to define with whom to share data– based on taggings
• e.g. everything tagged as „public“ is public• e.g. everything tagged as „forSteffen“ is accessible
for Steffen
– based on rules for access
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SEA: Data Model
• Ontological meta model
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Image understanding using ontologies
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
What is this?
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Solution
Better use context and
background knowledge
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Region-Based Image Labelling
1. Find semantically meaningful regions
2. Label them with concepts
3. Infer higher level annotations from initial labellings
4. Provide user-centred, semantic annotation
The overall aim is to improve the access to multimedia content.
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Image Labeling: 1. Initial, region-based
Output:• Segment Classification• Hypothesis set of
possible labels for image segments
• Degree of confidence
Scene classification
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Image Labeling: 1. Initial, region-based
Output of Person/Face Detection:
• Bounding boxes for detected persons/faces
• Degree of confidence
Scene classification
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Multi-Tier Image Model
Segments
Label hypotheses
Confidence values
Bounding boxes
Classification of picture
….
A1
A2
B2={l1,l2}
A1 over A2
A1 overlaps B2
…
Spatial & topological information
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Multimedia Reasoning
• Aim, now: – integrate available information towards – global, – consistent and – user-oriented annotation
• 3 tasks:– Consistency Checking– Region Merging– Generation of a higher-level, user-centered annotation
Current Focus
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Consistency Checking
Constraint Satisfaction Problem (CSP)
• Check that label(s) of a region are consistent wrt labels of neighboring regions
• Ideally:– Leaves one correct label per
region
• More often:– more than one label remains– decision in favor of highest
confidence values
Process consists of
1. Transformation of multi-tier description into a CSP
2. Application of constraint reasoning to solve the CSP
3. Computing the “best” labeling using the confidence values
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Constraint Satisfaction Problem (CSP)
Definition• Consists of set of variables and set of constraints
relating several variables• Each variable may have values from it’s domain• A constraint defines which values can be assigned to a
variable depending on the related variables• Standard methods exist to solve the CSP• Two steps:
– Consistency checking, i.e. removal of values from the domain that never satisfy the constraints
– Computation of full solutions using search algorithms(i.e. model generation)
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Constraint Satisfaction Problems
Example:• Variables:
x, y, z
• Domains:D(x) = {1, 2, 3}, D(y) = {2, 3, 4}, D(z) = {2, 3, 4, 5}
• Constraints:x >= y, y >= z
• After consistency checking: D(x) = {2, 3}, D(y) ={2, 3}, D(z) = {2, 3}
• Concrete Solutions (models):(2, 2, 2), (3, 2, 2), …
– Not all possible combinations of domain values are a solution!
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Image Labelling as a CSP
• Each segment s is transformed into a variable vs
• Initial Labellings Ls are the domains of the segment variables, D(vs)=Ls
• For each spatial relation type, a constraint sp-rel(v,w) is defined– the spatial constraints define which value
combinations are legal for the given relation• e.g. left-of(v,w):={(sea,sea),(sky,sky)}, but not (sky,sea)
• If two segments s,t are related with a spatial relation sp-rel, a corresponding spatial constraint sp-rel(vs,vt) is instantiated
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Initial image Segmentation Mask
Example
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Sea Sky Sand Person
Region 1 0.05 0.03 0.07 1.00
Region 2 0.28 0.42 0.30 0.00
Region 3 0.54 0.74 0.32 0.00
Region 4 0.79 0.54 0.43 0.08
Region 5 0.00 0.80 0.03 0.09
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
ConfidenceValues
Sea Sky Sand Person
Region 1 0.05 0.03 0.07 1.00Region 2 0.28 0.42 0.30 0.00Region 3 0.54 0.74 0.32 0.00Region 4 0.79 0.54 0.43 0.08Region 5 0.00 0.80 0.03 0.09
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SpatialRelations
ConfidenceValues
Sky can not be left of Sea
Sea Sky Sand Person
Region 1 0.05 0.03 0.07 1.00Region 2 0.28 0.42 0.30 0.00Region 3 0.54 0.74 0.32 0.00Region 4 0.79 0.54 0.43 0.08Region 5 0.00 0.80 0.03 0.09
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SpatialRelations
ConfidenceValues
Sky can not be left of Sea
Sea Sky Sand Person
Region 1 0.05 0.03 0.07 1.00Region 2 0.28 0.42 0.30 0.00Region 3 0.54 0.74 0.32 0.00Region 4 0.79 0.54 0.43 0.08Region 5 0.00 0.80 0.03 0.09
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SpatialRelations
ConfidenceValues
Sea Sky Sand Person
Region 1 0.05 0.03 0.07 1.00Region 2 0.28 0.42 0.30 0.00Region 3 0.54 0.74 0.32 0.00Region 4 0.79 0.54 0.43 0.08Region 5 0.00 0.80 0.03 0.09
Sea can not be above Sky
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
SpatialRelations
ConfidenceValues
Sea can not be above Sky
Sea Sky Sand Person
Region 1 0.05 0.03 0.07 1.00Region 2 0.28 0.42 0.30 0.00Region 3 0.54 0.74 0.32 0.00Region 4 0.79 0.54 0.43 0.08Region 5 0.00 0.80 0.03 0.09
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Definition of Spatial Constraints
• Spatial constraints form an integral part of the domain knowledge used for multimedia reasoning
• Currently they are explicitly defined by a domain expert• But:
– Seems not feasible for large amounts of concepts and relations– Preferably each constraint should be accompanied by a
confidence value, which can hardly be defined by an expert
• Idea:– Learn constraints from pre-annotated images– Allow for later refinement during run-time by user interaction.– Planned extension of M-OntoMat-Annotizer for this purpose– http://www.acemedia.org/aceMedia/results/software/m-ontomat-
annotizer.html
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
M-Ontomat-Annotizer
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Current Status:
• Use of segment classification
• Very recently: integration of person/face detection module
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Initial Evaluation
• An evaluation framework for region-based image labelling was defined within the aceMedia project.
• Ground Truth is defined on a grid-basis– a N x N grid is layered on top of each image– each cell is annotated with all depicted concepts
• For evaluation the segments of the segmentation, or the bounding boxes, are mapped to the respective cells.– For each concept it is counted how often
• the concept was found correctly, i.e. a correspondence between the segment label and a grid label is found
• the concept was found in general• the concept exists in the GT
• Based on these values precision and recall for each concept, and the overall process can be defined.
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Evaluation Results
• Since the method is based on content analysis modules, we evaluated the improvement reached by applying the constraint reasoning to the segment classification.
• First, precision, recall and the F-Measure were computed for the segment classification
• Then, for the CSP method applied to the initial labelling
• Finally the average improvement was calculated
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Evaluation Results
Concept
Precision
Recall
F
Sky 0.77 0.69 0.73
Sea 0.66 0.59 0.62
Sand 0.75 0.94 0.84
Person 0.33 0.65 0.44
Total 0.69 0.75 0.72
Segment Classification
Concept
Precision
Recall
F
Sky 0.78 0.91 0.84
Sea 073 0.53 0.62
Sand 0.85 0.97 0.9
Person 0.38 0.62 0.47
Total 0.76 0.82 0.79
Constraint Reasoning
• Set Up:– Evaluation with ~60 images– A 8x8 grid was used for the ground truth– The segmentation was set up to always produce 8 segments per image
• Results are promising, showing an 10% increase in average.• However, results in the overall performance are needed
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Next Steps
• Soft Constraint Reasoning– Fuzzy Constraints to integrate the confidence values into the
reasoning– Incremental Constraints to flexibly add constraints during
reasoning– both should provide for more robust results and lead to better
reduction of the initial label sets• Incorporation of a region merging step
– Would enable an iterative process and a knowledge-based segmentation
• Derivation of a higher-level annotation– Currently a simple combination of confidence values is applied
• the maximum degree for each concept is kept, and each concept is added to the final annotation
– later also relations and additional concepts should be inferred
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Conclusion1. Narrowing the Semantic Gap requires an
Integration of Multiple Techniques
2. Some of the techniques need not be very sophisticated – e.g. tagging
3. Some sophisticated techniques may not range very far – person recognition trained for my family doesn‘t
recognize Carsten
4. Different communities need to speak to each other
5. Large chances for the Semantic Web crowd!
Steffen Staab
ISWeb – Informationssysteme & Semantic Web
Bernhard Schüler
Thank You!
Sergej Sizov
Thomas Franz
Multimedia
Web ServicesP2P &
Complex Systems
Simon Schenk
S. Mir
F. S. ParreirasB. Tausch
The wonderful worldof ontologies@ISWeb
Klaas DellschaftOlaf GörlitzRabeeh Ayaz
Carsten Saathoff
C. Ringelstein
Steffen Staab
Richard Arndt