INFORMATIK Hendrik Lensch Hardware-Accelerated Silhouette Matching Hendrik Lensch, Wolfgang...

INFORMATIKINFORMATIK

Hendrik LenschHendrik Lensch

Hardware-AcceleratedHardware-AcceleratedSilhouette MatchingSilhouette Matching

Hendrik Lensch, Wolfgang Heidrich,Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seideland Hans-Peter Seidel

Max-Planck-Institut fMax-Planck-Institut für Informatik,ür Informatik,Saarbrücken (Germany)Saarbrücken (Germany)

Hendrik Lensch, Wolfgang Heidrich,Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seideland Hans-Peter Seidel

Max-Planck-Institut fMax-Planck-Institut für Informatik,ür Informatik,Saarbrücken (Germany)Saarbrücken (Germany)



OverviewOverview

• MotivationMotivation• Comparing SilhouettesComparing Silhouettes• Stitching and Combining TexturesStitching and Combining Textures• Results and ConclusionsResults and Conclusions

• MotivationMotivation• Comparing SilhouettesComparing Silhouettes• Stitching and Combining TexturesStitching and Combining Textures• Results and ConclusionsResults and Conclusions



Acquiring Real World ModelsAcquiring Real World Models

GeometryGeometry• 3D scanner3D scanner

GeometryGeometry• 3D scanner3D scanner

Texture dataTexture data• digital cameradigital camera

Texture dataTexture data• digital cameradigital camera

single sensor vs. multiple sensorssingle sensor vs. multiple sensors



3D – 2D Registration3D – 2D Registration

Find the camera setting for each 2D image.Find the camera setting for each 2D image.Find the camera setting for each 2D image.Find the camera setting for each 2D image.



TransformationsTransformations• to camera coordinates (extrinsic):to camera coordinates (extrinsic):

• to 2D image space (intrinsic):to 2D image space (intrinsic):

determine determine R, tR, t and and ff (6+1 dimensions) (6+1 dimensions)

TransformationsTransformations• to camera coordinates (extrinsic):to camera coordinates (extrinsic):

• to 2D image space (intrinsic):to 2D image space (intrinsic):

determine determine R, tR, t and and ff (6+1 dimensions) (6+1 dimensions)

tgxR

z

y

x

w

c

c

c

Camera ModelCamera Model

c

c

c y

x

z

f

v

u

v

u1

0

0



Similarity MeasureSimilarity Measure

Which features to investigate?Which features to investigate?• no color information on the modelno color information on the model• correspondence of geometric features hard to findcorrespondence of geometric features hard to find

Which features to investigate?Which features to investigate?• no color information on the modelno color information on the model• correspondence of geometric features hard to findcorrespondence of geometric features hard to find



Similarity MeasureSimilarity Measure

Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]

• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation

Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]

• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation



Distance Measure for Distance Measure for SilhouettesSilhouettes

Point-to-outline Point-to-outline distancesdistances

• slow because points on slow because points on the outline must be the outline must be determineddetermined

• speedup by distance speedup by distance mapsmaps

Point-to-outline Point-to-outline distancesdistances

• slow because points on slow because points on the outline must be the outline must be determineddetermined

• speedup by distance speedup by distance mapsmaps



11

00

intensityintensity

xx

differencedifference

11

00xx

Pixel-based Pixel-based Distance MeasureDistance Measure

Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images

• compute histogram compute histogram (hardware)(hardware)

• gives linear response to gives linear response to the displacementthe displacement




displacementdisplacement



1111

00

intensityintensity

xx

11

00


xx

Pixel-based Pixel-based Distance MeasureDistance Measure







displacementdisplacement



Approximation ofApproximation ofSquared DistancesSquared Distances

Use smooth transitionsUse smooth transitions• blur imagesblur images

• integrate squared integrate squared differencesdifferences

• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost






Approximation ofApproximation ofSquared DistancesSquared Distances







1111

00

intensityintensity

xx

1111

00

intensityintensity

xx

11

00


xx


11

00xx

filtersizefiltersize



Non-linear Optimization Non-linear Optimization

Downhill Simplex Method [Press 1992] Downhill Simplex Method [Press 1992] • works for works for NN dimensions dimensions• no derivatives no derivatives • easy to controleasy to control

Downhill Simplex Method [Press 1992] Downhill Simplex Method [Press 1992] • works for works for NN dimensions dimensions• no derivatives no derivatives • easy to controleasy to control



Simplex Method in 3D Simplex Method in 3D

original simplexoriginal simplexoriginal simplexoriginal simplex

reflection and/orreflection and/orexpansionexpansionreflection and/orreflection and/orexpansionexpansion

shrinkingshrinkingshrinkingshrinking

random random perturbationperturbationrandom random perturbationperturbation



Hierarchical OptimizationHierarchical Optimization

• optimize on low resolution firstoptimize on low resolution first• restart optimization to avoid local minimarestart optimization to avoid local minima• switch to higher resolutionswitch to higher resolution• mesh resolution can be adaptedmesh resolution can be adapted

• optimize on low resolution firstoptimize on low resolution first• restart optimization to avoid local minimarestart optimization to avoid local minima• switch to higher resolutionswitch to higher resolution• mesh resolution can be adaptedmesh resolution can be adapted



Starting Point GenerationStarting Point Generation

• set camera distance set camera distance ttzz depending on object size depending on object size

• setset ttxx and and ttyy to zero to zero• select 48 sample rotationsselect 48 sample rotations• run optimization for each of the samplesrun optimization for each of the samples

(40 evaluations)(40 evaluations)• select top 5 resultsselect top 5 results• restart optimization (200 evaluations)restart optimization (200 evaluations)• take best result as starting pointtake best result as starting point

• set camera distance set camera distance ttzz depending on object size depending on object size

• setset ttxx and and ttyy to zero to zero• select 48 sample rotationsselect 48 sample rotations• run optimization for each of the samplesrun optimization for each of the samples

(40 evaluations)(40 evaluations)• select top 5 resultsselect top 5 results• restart optimization (200 evaluations)restart optimization (200 evaluations)• take best result as starting pointtake best result as starting point



Texture StitchingTexture Stitching

projective texture mappingprojective texture mapping assign one image to each triangle assign one image to each triangle • triangle visible in image? (test every vertex)triangle visible in image? (test every vertex)• select best viewing angleselect best viewing angle• discard data near depth discontinuities discard data near depth discontinuities

projective texture mappingprojective texture mapping assign one image to each triangle assign one image to each triangle • triangle visible in image? (test every vertex)triangle visible in image? (test every vertex)• select best viewing angleselect best viewing angle• discard data near depth discontinuities discard data near depth discontinuities



Blending Across Blending Across Assignment BordersAssignment Borders

• find border verticesfind border vertices

• release all triangles release all triangles around them around them

• assign boundary assign boundary vertices to best regionvertices to best region

• assign alpha-values for assign alpha-values for each region each region

– 1 to vertices 1 to vertices included in the regionincluded in the region

– 0 to all others.0 to all others.

• find border verticesfind border vertices

• release all triangles release all triangles around them around them

• assign boundary assign boundary vertices to best regionvertices to best region

• assign alpha-values for assign alpha-values for each region each region

– 1 to vertices 1 to vertices included in the regionincluded in the region

– 0 to all others.0 to all others.



Entire TextureEntire Texture



Results and ConclusionsResults and Conclusions

Problems solved:Problems solved:• automatic texture registration (automatic texture registration (R, t, fR, t, f))• view-independent texture stitchingview-independent texture stitching• blending across assignment boundariesblending across assignment boundaries• rough manual alignment helps (speedup, failures) rough manual alignment helps (speedup, failures)

Further problems:Further problems:• extract purely diffuse part of textureextract purely diffuse part of texture• generate texture where data is missinggenerate texture where data is missing

Problems solved:Problems solved:• automatic texture registration (automatic texture registration (R, t, fR, t, f))• view-independent texture stitchingview-independent texture stitching• blending across assignment boundariesblending across assignment boundaries• rough manual alignment helps (speedup, failures) rough manual alignment helps (speedup, failures)

Further problems:Further problems:• extract purely diffuse part of textureextract purely diffuse part of texture• generate texture where data is missinggenerate texture where data is missing



Questions?Questions?

visit us at visit us at

www.mpi-sb.mpg.dewww.mpi-sb.mpg.de

visit us at visit us at

www.mpi-sb.mpg.dewww.mpi-sb.mpg.de

INFORMATIK Hendrik Lensch Hardware-Accelerated Silhouette Matching Hendrik Lensch, Wolfgang...

Documents

Transcript of INFORMATIK Hendrik Lensch Hardware-Accelerated Silhouette Matching Hendrik Lensch, Wolfgang...