1

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Image Semantic Coding using OTB

Marie Liénou - Marine CampedelTélécom ParisTech

July 2009

2

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

SUMMARY

Semantic CodingOTB tool

COCNotion of

semantic CodingA promising

approach

Development of an OTB toolConclusion and

perspectives

3

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

COC = COmpetence Centre…

Tripartite agreement between CNES – DLR and Télécom ParisTech

Signed in June 2005

Goal : joint action on image understanding SAR/Optical, HR and VHR, temporal series Feature extraction, modeling, indexing, compression,

(interactive) classification, interpretation, knowledge representation, reasoning, …

Means ~ 4 new phds / year ~10 permanent researchers partially involved financial support for specific actions (studentships, engineers,

post-docs)

4

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Image Semantic coding

Semantic Coding

Compression Reduce data while

ensuring informational content

MeaningUnderstandingInterpretationImage to text?

[Barnard et al., 2003 ; Jeon et al., 2003][Li et Bretschneider, 2006]

Goal: find an image representation able to capture the contained semantics Idea: use text indexing approach + active learning

5

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Image Semantic coding

Feature extraction

Quantization

« visual words »

Indexing Mining

Active learning

Visual interaction Manual annotation

Where is semantics?

Automatic annotation

6

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Image Semantic coding vs KIM

« Design and evaluation of HMC for Image Information Mining » Daschiel and Datcu IEEE transaction on multimedia, vol 7, no6, dec. 2005

7

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

A promising approach

Feature extraction Segmentation, arbitrar regions “Classical” signature: color, texture, shape descriptors Experiments: intensity mean and variance in each spectral band

Quantization K-Means: each estimated cluster corresponds to one “visual word” K estimated using MDL (Minimum Description Length) descriptor

Bag-of-words signature for semantics identification Count visual words on image regions which will be annotated Normalize (tf-idf)

Exploitation using machine learning (SVM, LDA)

8

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Marie Lienou PhD work (march 2009) Tested on several VHR (multispectral) images Compared to other classification approachs (GMM, SVM)

Recognition accuracy demonstrated for “semantically complex” classes Ex: “urban area”

LDA = fast + does not need negative examples

A promising approach

Feature extraction

QuantizationClassificationSVM, LDA

Count words

Feature extraction

ClassificationGMM, SVM

Majority rule

Annotations

Low level annotations

Visual word production

9

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

OTB tool: cocSemanticCoding

Feature extraction Vectorial image with as many components as feature dimension Exploitation of OTB extractors at each pixel

Quantization Use of K-Means filter

Bag-of-words signature Count visual words on image regions which will be annotated Normalization (tf-idf)

Learning from manual annotation Fluid interface facilities Learn LDA from only target samples Learn SVM from target samples and counter examples Classify the whole image Iterate

10

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

OTB tool: cocSemanticCoding

11

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

OTB tool: cocSemanticCoding

12

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

OTB tool: cocSemanticCoding

13

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

14

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

15

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

16

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Learning and classification toolsLDA on occurrence dataSVM on TFiDF data (features)Both results can be obtained with same labeling for comparisonDifficulty for the user : compute features adapted to the underlying semantics

17

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Conclusion

OTB useful features Vectorial image representation Great diversity of available filters (extractors, classifiers)

New = LDA classifier + estimator Visualization tools

cocSemanticCoding tool availability www.tsi.enst.fr/~campedel/ will be updated

Necessity to valorize research results Engineering process (C++ programming) Not easy but OTB is a nice initiative to help researchers In the future: centralize processing tools (in OTB) + easy their

exploitation (documentations, interfaces)

18

Com

pe t

ence

Cen

tre

on I

nfo

rmat

ion

Ext

r act

ion

an

d I

mag

e U

nd

erst

a nd

ing

for

Ear

th O

bs e

rvat

ion

Perspectives

Other COC tools should be integrated in cocSemanticCoding MDL to estimate of visual words number new feature extractors (QMF-based texture descriptors) Feature selection Complete relevance feedback framework

New approaches for image interpretation From semantics to knowledge? Knowledge engineering: modeling (ontologies) + reasoning Several works on characterizing relations between identified

concepts and/or image objects