Methods for 3D Shape Matching and Retrieval

Marcin Novotni & Reinhard KleinMarcin Novotni & Reinhard KleinUniversity of BonnUniversity of Bonn

Computer Graphics GroupComputer Graphics Group

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Our Aim #1Our Aim #1Our Aim #1Our Aim #1

Given an example:Given an example:

,, ,…

Find the most Find the most

similar object(s)similar object(s)

in a databasein a database

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

MotivationMotivationMotivationMotivation

Lots of 3D archives:Lots of 3D archives: WWW WWW Proprietary databasesProprietary databases ......

Search engines for data:Search engines for data: Text, 2D images, music (MIDI), …Text, 2D images, music (MIDI), … Emerging since 1998 for 3DEmerging since 1998 for 3D

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Our Aim #2Our Aim #2Our Aim #2Our Aim #2

Direct matching Direct matching AlignmentAlignment Establishing correspondencesEstablishing correspondences

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

MotivationMotivationMotivationMotivation

Partial matching/retrievalPartial matching/retrieval

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

MotivationMotivationMotivationMotivation

Partial matching/retrievalPartial matching/retrievalStatistical shape analysisStatistical shape analysisMorphingMorphingTexture transferTexture transferRegistrationRegistration

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

General ProblemGeneral ProblemGeneral ProblemGeneral Problem

Abstract representationAbstract representation facilitating: facilitating:

identification of salient features of 3D identification of salient features of 3D objectsobjects

description of featuresdescription of featurescomparison (matching)comparison (matching)

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

OverviewOverviewOverviewOverview

Matching for 3D Shape RetrievalMatching for 3D Shape Retrieval

Correspondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Matching for Matching for 3D Shape 3D Shape RetrievalRetrieval

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

General ProblemGeneral ProblemGeneral ProblemGeneral Problem

We need a We need a DescriptorDescriptor

D : → D( )

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

d( , )

d( , )

General ProblemGeneral ProblemGeneral ProblemGeneral Problem

We need a We need a Distance MeasureDistance Measure

:=

D( )

D( )

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

d( , ) d( , )

General ProblemGeneral ProblemGeneral ProblemGeneral Problem

We need a We need a Distance MeasureDistance Measure : : Close to (application driven) notion of resemblanceClose to (application driven) notion of resemblance Computationally cheap and robustComputationally cheap and robust

d( , )≤ ≤

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

Feature vectorsFeature vectors

XXii : 3D Zernike Descriptors : 3D Zernike Descriptors [Canterakis ’99, Novotni & Klein ’03, ’04][Canterakis ’99, Novotni & Klein ’03, ’04]Distance Measure: Euclidean DistanceDistance Measure: Euclidean Distance

D( ) ≡x1

xn

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

Retrieval performance Retrieval performance [Novotni & Klein ‘03 ’04][Novotni & Klein ‘03 ’04]

Slightly better than [Funkhouser et al. ’02]Slightly better than [Funkhouser et al. ’02]

Object class dependent performance!Object class dependent performance! Class dependent coefficient importance!Class dependent coefficient importance!

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D 3D Zernike DescriptorsZernike Descriptors3D 3D Zernike DescriptorsZernike Descriptors

Importance

Coeff No. (Frequency)

ChairsFaces Airplanes

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

Relevance feedback:Relevance feedback: User selects relevant / irrelevant itemsUser selects relevant / irrelevant items Distance measure is tunedDistance measure is tuned

Learning Machines:Learning Machines: SVM (Support vector machines) [Vapnik ‘95]SVM (Support vector machines) [Vapnik ‘95] One class SVM [SchOne class SVM [Schöölkopf et al. lkopf et al. ’’9999]] (K)BDA ((Kernel) Biased Discriminant (K)BDA ((Kernel) Biased Discriminant

Analysis) [Zhou et al. ‘01]Analysis) [Zhou et al. ‘01]

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

CorrespondenCorrespondence Matchingce Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Geometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity Estimation

Idea Idea [Novotni & Klein 2001][Novotni & Klein 2001]:: Definition of „geometric“ similarity in terms of Definition of „geometric“ similarity in terms of

a geometric distancea geometric distance

Intuitive, simple, robust.Intuitive, simple, robust.

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Geometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity Estimation

6.786.78 8.858.85 30.2930.29 38.0938.09 67.5367.53Normalized Normalized volumetric volumetric

errorerror0.000.00

Database objects

example

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Geometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity Estimation

Classification by user set thresholdClassification by user set threshold

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Geometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity Estimation

Measures deformation magnitudeMeasures deformation magnitude

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

?

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

?

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

?

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

?

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Ideally: dense mappingIdeally: dense mapping

?

22: f

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Ideally: dense mappingIdeally: dense mapping

Deformation by mapping semanticsDeformation by mapping semantics

[D’Arcy Thompson 1917: On Growth and Form ]

22: f

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Ideally: dense mappingIdeally: dense mapping

Easier: mapping Easier: mapping salient pointssalient points Curvature extremesCurvature extremes Corners (Harris points in 2D)Corners (Harris points in 2D) Etc…Etc… Scale space extremesScale space extremes

22: f

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Ideally: dense mappingIdeally: dense mapping

Easier: mapping Easier: mapping salient pointssalient points Curvature extremesCurvature extremes Corners (Harris points in 2D)Corners (Harris points in 2D) Etc…Etc… Scale space extremesScale space extremes

22: f

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Scale Space extremes Scale Space extremes [Lindeberg ‘94][Lindeberg ‘94]

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

We have:We have:Salient pointsSalient points

Spatial positionSpatial position Size of local blobsSize of local blobs

How to match???How to match???

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Criteria for correspondences:Criteria for correspondences:

SimilarSimilarLocal geometriesLocal geometriesConstellations of pointsConstellations of points

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Criteria for correspondences:Criteria for correspondences:

SimilarSimilarLocal geometriesLocal geometriesConstellations of pointsConstellations of points

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Local descriptionLocal description

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Local descriptionLocal description

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Assumption:Assumption:

Similar local descriptors Similar local descriptors

Similar local geometriesSimilar local geometries

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Criteria for correspondences:Criteria for correspondences:

SimilarSimilarLocal geometriesLocal geometriesConstellations of pointsConstellations of points

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Similar constellations of pointsSimilar constellations of points Smooth mappings leave constellations Smooth mappings leave constellations

consistentconsistent

IdeaIdea Constellations are consistent if mapping is Constellations are consistent if mapping is

smoothsmooth

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Similar constellations of pointsSimilar constellations of pointsIdea: Idea:

Constellations are consistent if mapping is Constellations are consistent if mapping is smoothsmooth

Thin Plate Spline interpolation [Brookstein ’89]Thin Plate Spline interpolation [Brookstein ’89]

minimize:minimize:2( ) ( )

d

I f f d x x

Total curvature

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

minimize:minimize:

Minimizer (Minimizer (Thin Plate SplineThin Plate Spline interpolator): interpolator):

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

2( ) ( )d

I f f d x x

1

( )( ) ( )

( )

xx lkyy

k k

wf

wf

k

xf x Ax t x x

x

Affine part Nonlinear deformation

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

minimize:minimize:

Minimizer (Minimizer (Thin Plate SplineThin Plate Spline interpolator): interpolator):

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

2( ) ( )d

I f f d x x

1

( )( ) ( )

( )

xx lkyy

k k

wf

wf

k

xf x Ax t x x

x

2( ) logr r r 2D Thin Plate Spline

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

1

( )( ) ( )

( )

xx Nkyy

k k

wf

wf

k

xf x Ax t x x

x

minimize:minimize:

Minimizer (Minimizer (Thin Plate SplineThin Plate Spline interpolator): interpolator):

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

2( ) ( )d

I f f d x x

Can be computed by a (N+4)x(N+4) matrix

inversion

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Find (sub)sets of correspondences:Find (sub)sets of correspondences: Small local descriptor distancesSmall local descriptor distances Small deformation energySmall deformation energy

Hierarchical pruning and clusteringHierarchical pruning and clusteringUsing:Using:

Local descriptorsLocal descriptors Geometrical constellation consistencyGeometrical constellation consistency

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

New avenuesNew avenues::Local Descriptions for retrievalLocal Descriptions for retrievalOnline Learning for local Online Learning for local

descriptionsdescriptionsDense matching from salient pointsDense matching from salient pointsEtc.Etc.

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Danke,Danke,DFG!DFG!

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Basis functions in the unit sphere:Basis functions in the unit sphere:

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

SH on the sphere

( , , ) ( ) ( , )m mnl nl lZ r R r Y

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Basis functions in the unit sphere:Basis functions in the unit sphere:

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

SH on the sphere

Function of the radius

( , , ) ( ) ( , )m mnl nl lZ r R r Y

Rotation invariant!

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Basis functions in the unit sphere:Basis functions in the unit sphere:

3D Zernike Moments 3D Zernike Moments [Canterakis ‘99]:[Canterakis ‘99]:

: ,m mnl nlf Z

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

( , , ) ( ) ( , )m mnl nl lZ r R r Y

Object function, e.g. voxel grid

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

3D Zernike Descriptors:3D Zernike Descriptors: Amplitudes of the Zernike decompositionAmplitudes of the Zernike decomposition

Rotation invariantRotation invariant

1

: ,

lnllnl

nl

lnl

F

0

0

. . ( ) is even

n N

l n

s t n l

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Basis functions in the unit sphere:Basis functions in the unit sphere:

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

SH on the sphere

( , , ) ( ) ( , )m mnl nl lr R r Y

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Basis functions in the unit sphere:Basis functions in the unit sphere:

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

SH on the sphere

Function of the radius

( , , ) ( ) ( , )m mnl nl lr R r Y

Rotation invariant!

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Basis functions in the unit sphere:Basis functions in the unit sphere:

3D Zernike Moments 3D Zernike Moments [Canterakis ‘99][Canterakis ‘99]::: ,m m

nl nlf Z

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

( , , ) ( ) ( , )m mnl nl lr R r Y

Object function, e.g. voxel grid

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

3D Zernike Descriptors:3D Zernike Descriptors: Amplitudes of the Zernike decompositionAmplitudes of the Zernike decomposition

Rotation invariantRotation invariant

1

: ,

lnllnl

nl

lnl

F

0

0

. . ( ) is even

n N

l n

s t n l

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

For N=22 : 155 floats as search keyFor N=22 : 155 floats as search key

Timings (1.8 GHz Pentium):Timings (1.8 GHz Pentium):Voxelization: 0.3 – 10.0 sec / objectVoxelization: 0.3 – 10.0 sec / objectComputation: 0.2 sec / objectComputation: 0.2 sec / objectRetrieval (1814 objects): 0.3 secRetrieval (1814 objects): 0.3 sec

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

Retrieval performance Retrieval performance [Novotni & Klein ’04][Novotni & Klein ’04]

Slightly better than [Funkhouser et al. ’02]Slightly better than [Funkhouser et al. ’02]

Object class dependent performance!Object class dependent performance! Class dependent coefficient importance!Class dependent coefficient importance!

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D 3D Zernike DescriptorsZernike Descriptors3D 3D Zernike DescriptorsZernike Descriptors

Importance

Coeff No.

ChairsFaces Airplanes

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

3D Zernike functions [Canterakis ‘99]3D Zernike functions [Canterakis ‘99]

are polynomials such that are are polynomials such that are orthonormal within the unit ballorthonormal within the unit ball

( , , ) : ( ) ( , )m mnl nl lZ r R r Y

mnlZnlR

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

3D Zernike functions [Canterakis ‘99]3D Zernike functions [Canterakis ‘99]

are polynomials such that are orthonormal are polynomials such that are orthonormal within the unit ballwithin the unit ball

3D Zernike Moments:3D Zernike Moments:

( , , ) : ( ) ( , )m mnl nl lZ r R r Y

mnlZnlR

: ,m mnl nlf Z

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

3D Zernike Descriptors:3D Zernike Descriptors: Amplitudes of the Zernike decompositionAmplitudes of the Zernike decomposition

Rotation invariantRotation invariant

1

: ,

lnllnl

nl

lnl

F

0

0

. . ( ) is even

n N

l n

s t n l

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

For N=20 : 121 floats as search keyFor N=20 : 121 floats as search key

Timings (1.8 GHz Pentium):Timings (1.8 GHz Pentium):Voxelization: 0.3 – 10.0 sec / objectVoxelization: 0.3 – 10.0 sec / objectComputation: 0.2 sec / objectComputation: 0.2 sec / objectRetrieval (1814 objects): 0.3 secRetrieval (1814 objects): 0.3 sec

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

Retrieval performance Retrieval performance [Novotni & Klein ’04][Novotni & Klein ’04]

Slightly better than [Funkhouser et al. ’02]Slightly better than [Funkhouser et al. ’02]

Object class dependent performance!Object class dependent performance! Class dependent coefficient importance!Class dependent coefficient importance!

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Matching should be:Matching should be:

Independent of topologyIndependent of topologyRobustRobustSuitable for partial matchingSuitable for partial matching

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Local descriptionLocal description Local shape histograms Local shape histograms

Not rotation invariantNot rotation invariant

Rotation invarianceRotation invariance

Amplitudes of the Fourier Amplitudes of the Fourier TransformTransform

( , )r

1ˆ ( , ) ( , ), ip

Sr p r e

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

155 floats as search key155 floats as search key

Retrieval (1814 objects): 0.3 secRetrieval (1814 objects): 0.3 sec

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Stuff to rememberStuff to remember::

Salient points simplify the problemSalient points simplify the problemSmooth mapping iff consistent Smooth mapping iff consistent

constellationsconstellations

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Stuff to rememberStuff to remember::

Salient points simplify the problemSalient points simplify the problem VolumetricVolumetric On the surfaceOn the surface

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Stuff to rememberStuff to remember::

Salient points simplify the problemSalient points simplify the problemSmooth mapping iff consistent Smooth mapping iff consistent

constellationsconstellations

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

New avenuesNew avenues::Local Descriptions for retrievalLocal Descriptions for retrieval

Retrieval by part selection & recognitionRetrieval by part selection & recognitionRetrieval from large scenesRetrieval from large scenes

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

New avenuesNew avenues::Local Descriptions for retrievalLocal Descriptions for retrievalOnline Learning for local Online Learning for local

descriptionsdescriptionsAdopting pattern recognition methodsAdopting pattern recognition methods

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

New avenuesNew avenues::Local Descriptions for retrievalLocal Descriptions for retrievalOnline Learning for local Online Learning for local

descriptionsdescriptionsDense matching from salient pointsDense matching from salient points

Morphing, registration, object statisticsMorphing, registration, object statistics

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Our Aim #2Our Aim #2Our Aim #2Our Aim #2

Direct matching Direct matching AlignmentAlignment

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Scale space extremes Scale space extremes [Lindeberg ‘94][Lindeberg ‘94] Blob detection by Blob detection by localizing extremes of Laplacian … localizing extremes of Laplacian … … … in scale and spacein scale and space

Size of the blob

Position of the blob

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Maxima of Laplacian over scalesMaxima of Laplacian over scales

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

Spatial maximaSpatial maxima