Interactive Point-based Modeling of Complex Objects from Images
Pierre Poulin (a,b)Marc Stamminger (a,c)François Duranleau (b)
Marie-Claude Frasson (a)George Drettakis (a)
(a) REVES, INRIA Sophia Antipolis(b) DIRO, Université de Montréal
(c) University of Erlangen
Modeling Complex Objects
Modeling Complex Objects
• High visual complexity
• Time consuming
• Algorithms for specialized objects– e.g., plants, mountains, etc.
• Adaptive rendering
• Many applications need such objects
Key Observations
• Extracting complex models from photos is a very powerful approach
• Point-based representation is very effective for complex models– Efficient display and storage
• User interaction is beneficial when extracting quality models– Specify where details are needed
– Resolve some ambiguities
Image-based Point Modeling
• Images are very flexible– Reality-based (photos)– Acquisition is easy
Image-based Point Modeling
• Points are very flexible– Fast rendering (hardware support)– Adaptive rendering for interactive display
Stamminger
Image-based Point Modeling
• Points are very flexible– Hierarchical organization and levels of detail
Q-splat
Image-based Point Modeling
• Points are very flexible– Visual quality– Many recent advances
Deussen
Automatic ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
User
new imagesrequantizerecalibrate
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
User
color comparisonsplausibility threshold
new depth mapszone of interest
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
Userrevalidate the pointsrequest more pointsdecimate the points
jitter the pointssample with patterns
hole filling
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
User
undo changesremove pointsadd polygons
Interactive Reconstruction
• Interactive display– 6 M points/sec. on a PIII 1GHz with GeForce3
• Efficient reconstruction algorithm– Test more than 1K points/sec.
• Simple and intuitive controls– Direct interaction with the points
Computer Vision Contributions
• 3D scanners
• Structured light
• Stereo – N-views
• Shape-from-X
• Volumetric
Volumetric Reconstruction
• Voxel coloring and Space carving– If a voxel is impossible, carved out of object– Silhouettes, transparency, shading– Photo-consistency
SeitzKutulakos
Image-based Polygon Modeling
• Academic: Façade, Rekon, Reality
• Industry: RealViz, Canoma, Photomodeler
Façade
Image-based Polygon Modeling
• Small polygonal scene (30-100 polygons)
• Extracted textures and illumination
Boivin
Input Images (4/14)
Input Images
• Digital camera: Canon EOS-DS30
• 1080x720 and 2166x1440
• Fixed aperture and shutter speed
• Try not to change zoom
• OpenGL and ray traced test scenes
Camera Calibration
Camera Calibration
• ImageModeler from RealViz
• Fiduciary marks placed around the object
• Interactive system
• Intrinsic and extrinsic camera parameters
3D Zone of Interest
Initial Random Points
Initial Random Points
• Generated randomly within the envelope
• More specific patterns discussed later
• Projection of a point in each photo
• Gather colors
Color Comparison
• Euclidean distance– RGB, CIE xy, CIE Luv, CIE Lab
– Speed vs. accuracy
• Color quantized images– Precomputed (ppmquantall or more sophisticated)
– Quantization only on projected zone of interest
– 32 to 128 colors
– Reduce shading variations
– Efficient test for color equality
C: 25%B: 50%
Dominant Color
A: 100%
Plausibility
100%33%
with visibility
Random Points with Depth Maps
Depth Maps
• Computed from the current set of points
• Updated on user demand
• With depth maps, can raise the plausibility threshold
• Generate more points within the object
• Re-evaluation of previously generated points
Clean-up Points
Clean-up Points
• In general– Increase color threshold and re-evaluate
• With good depth maps– Project in each image– Reject if point visible and color too different
Generate More Points
Generate More Points
• Randomly
• Stratified sampling based on voxels
• Point decimation based on voxels
Guide the Points
Guide the Points
• Smaller 3D sphere of interest– Generate more points– Eliminate all points
• 3D flood fill for branching patterns
• Patterns for planar surfaces
• Patterns for boundary surfaces
Filling with no Leaves
Filling with Leaves
Jitter the Points
Reprojection
Stepping through it again
Results
Scene Images Resolution Colors Points
Fruit bowl 13 512x512 - -
Soldier 13 2160x1440 64 118K
Snack 8 1440x960 64 120K
Ficcus 13 2160x1440 64 150K
Synthetic Fruit Bowl
ray tracingcolor points reprojection
Toy Soldier
color pointscolor points reprojection
Snack
Snack
Ficcus
Conclusions
• Point-based reconstruction of complex objects from images
• Tight integration– 3D color point representation– User-driven and/or automatic reconstructions– Interactive display
• Flexible to integrate most advances in computer vision
Findings
• First steps are encouraging, but objects are still of limited realism
• Information in photos is inspiring, but also difficult to analyse correctly
• How many things in a pixel?
• How many pixels and colors for an object?
Future Work
• Video sequences
• High dynamic range photos
• Shadows and shading in color comparison
• Extraction of limited BRDFs
• 3D texture synthesis of materials
Questions
• Did you…
• Is it…
• Can you…
• When…
• What…
• Where…
User Interaction in Modeling
• Specify regions of interest, thresholds, validity
• Control the visual quality
• Iterative refining process
• Guide the solution
• Automatic or interactive process
• Interactive display (6 M points/sec. GeForce3)
Image-based Point Modeling
• Difficulties with points – Visibility
• Holes in surfaces, size of points
• Filtering the representation and the texture
• Not our goal to fix these difficulties for now
LOD in Graphics
• Environment maps
• Billboards
• Textured polygons
• Layer-depth images
• Light field / lumigraph
3D Scanners
• Very good results in general
• Size of the scanner wrt object
• Costs
• Fixed illumination
Stereo - N views
• Camera calibration• Epipolar constraints• Color matching• 3D position and color• Difficulties
– Holes and occlusions– Sharp edges, noise, shading
• Infinity of shapes in general• Targeted for object recognition and collision avoidance• Only recently goal of photo-realism
Shape-from-X
• Silhouettes
• Shadows
• Focus/defocus
• Motion
• Shading
• etc.
Top Related