ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces.

Digital Days 29/6/2001

ISTORAMA: A Content-Based Image Search Engine andHierarchical Triangulation of 3D Surfaces.

Dr. Ioannis Kompatsiaris

Centre for Research and Technology Hellas

Informatics and Telematics Institute

Thermi-Thessaloniki, Greece

[email protected]


Outline• Introduction• Istorama architecture• K-Means with Connectivity Constraint Algorithm (KMCC)• Demo• Object/model based coding• Adaptive Triangulation and Progressive transmission• Reduced pyramid - quincunx sampling• Experimental results• Conclusions


Need for efficient image search

• Huge number of images or databases of images

• Highly visual and graphical nature of the Web

• Text descriptors are not always efficient

• Greater flexibility with “content-based” access

• Queries which are more natural to humans


Proposed approach

• Usually a description, a “signature” or a set indexes is created for the whole image

• Images usually contain different objects• Proposed approach: the image is first separated

into objects (segmentation)• Descriptors are created for each object• The user can search for a specific object

contained in images


ISTORAMA architecture

Server

World Wide Web

Data BaseJDBC

Java Data Base Connection

User

PHP

Crawler - Spider

Indexing - Retrieval Algorithms


The K-Means with Connectivity Constraint Algorithm (KMCC) I

• Based on K-Means algorithm• K-Means does not take into account spatial

information• In KMCC, the spatial proximity of each region is

also taken into account by defining a new spatial center and by integrating the K-Means with a component labeling procedure

• Automatic correction of the number of regions KK


The K-Means with Connectivity Constraint Algorithm (KMCC) II

• Step1 K-Means is performed • Step2 Spatial centers are calculated

• Step3 Generalised distance

• Step 4 Component labeling LL connected regions

kCI

k

k

Sk

I AAkD

CSpCIpIp

22

1)(),(

kCS


The K-Means with Connectivity Constraint Algorithm (KMCC) III


Object descriptors

• Color, texture and spatial characteristics

• Color: histogram, 8 bins

• Spatial: (centroid),

• Shape: area, eccentricity

where λ1, λ2 are the two first eigenvalues

kCS

2

11


Experimental Results (Synthetic)


Experimental Results


Experimental Results (Claire)

Facial region

Moving object

Original sequenceFrames 1-

10

Segmentation


Experimental results (Claire)


Experimental results (table-tennis)

Original sequenceFrames 1-10


Experimental results (table-tennis)

Segmentation

Moving objects


Experimental results (Akiyo+Foreman)

Facial region

Facial region


10


10


Conclusions

• K-means with spatial proximity algorithm• Multiple features segmentation• Higher order segmentation• Correspondence of objects between consequent

frames• Max-min criterion for automatic regularisation

parameters


Future work

• Use of texture

• Indexing of video

• Integration with text descriptors


• Triangular meshes of high quality are used in:

• Computer Aided Design • 3D representation of objects

(e.g. archaeological artifacts)• Animation and visual simulation• Entertainment (computer games)• Digital Terrain Modelling

Introduction


Object/model-based coding


Compression of finely detailed surfaces is necessary for:

• computation

• storage

• transmission

• display efficiency

Adaptive triangulation


• Early, coarse approximations are refined though additional bits

Progressive transmission


• Vertices removal and retriangulation [Schroeder] [Cohen]

• General mesh optimization process/function [Hoppe]

• Multiresolution analysis (MRA) [Lounsbery]

• Wavelets [Schroeder] [Gross]

• Progressive transmission [Schroeder] [Hoppe]

• Generalized triangle mesh representation [Deering]

Background


Properties of the algorithm

• Efficient compression of the wireframe information• Simplification of the wireframe by adaptive

triangulation• Progressive transmission of the wireframe

information• Prioritised transmission of the wireframe• Straightforward correspondence between

successive scales


Input surfaces

• Surface represented as a parametric function

in the parametric space

• determined by the position of a set of control points or nodes

• It allows for arbitrary, possibly closed wire-frame surfaces to be defined.

TvuzvuyvuxvuP ),(),,(),,(),(

2R

Tklklkl zyxlkr ,,),(


Input surfaces

• The filters are applied to the 2D parametric representation of the surface as though it were a 2D image with intensity equal to

• Such surfaces include also:• depth images estimated from stereo pairs and• every surface that is homomorphic to a plane,

cylinder or torus

),( vuP

),( vuz


Block diagram of the proposed procedure


Reduced pyramid with quincunx sampling matrix


Corresponding triangulation


Optimal filters

• Optimal filters are determined by their Fourier transform:

• where is the power spectral density.

• Alternatively may be determined by the equation:

1,,1,0,1

021

Nr

eeeG M

iqj

rM

jrj

ri

T Mw

ww

Mk)(ig

wjr e

tktMMkpMtk

,)()(r

irir RgR


Optimum bit allocation

• bits/vertex is assumed to be transmitted

• bits/vertex are allocated to each level using

• is the sum of error variances

BB rr 2

2

B

rB

2

r

ir

0

22


Error prioritization

• The prediction errors corresponding to all predicted vertices are calculated and sorted with the vertices corresponding to higher errors being put first on the list

Higher Errors

Lower Errors


Entropy estimation

• Entropy coding is used• The number of bits needed for error transmission

is the entropy of the errors • Using the quincunx sampling geometry at the

receiver, there is no need to transmit the exact co-ordinates of the position of each transmitted vertex

• The final cost of the transmission is the sum of the error entropy and the position entropy


Adaptive Triangulation Procedure

• Synthesis stage of the QMVINT pyramid

• The vertex along with the vertices used to predict it are added to the mesh

• Handling of cracks

• Triangulation of the next vertex

)()(ˆ)( )()()( rk

rk

rk PerrPIPI

)(rkP


Adaptive triangulation procedure


Experimental results

• Original dense depth map and surface of the “Venus” data



• 2569 vertices and 4006 triangles at level 2 MSE = 1.30


Conclusions

• Hierarchical representation of 3D surfaces using 3D adaptive triangular wireframes

• The variance of the error transmitted is minimised and therefore results to optimal compression of the wireframe information

• It produces a hierarchy where coarse meshes are as similar to their finer versions as is possible


Conclusions

• The triangulation algorithm is integrated with a bit allocation procedure

• The number of nodes and triangles of the wireframe as well as the information needed for the transmission or storage of the wireframe are reduced simultaneously using a unified approach (QMVINT filtering)

• Precise correspondence between triangles at each level is achieved


Future work

• Expansion and application directly to 3D surfaces

• Estimation of filters

ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces.

Documents

Transcript of ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces.