lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I...
Transcript of lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I...
![Page 1: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/1.jpg)
3D photography
Digital Visual Effects, Spring 2009g , p gYung-Yu Chuang2009/5/142009/5/14
with slides by Szymon Rusinkiewicz, Richard Szeliski, Steve Seitz and Brian Curless
![Page 2: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/2.jpg)
3D photography
• Acquisition of geometry and material
![Page 3: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/3.jpg)
Range acquisition
![Page 4: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/4.jpg)
Range acquisition taxonomy
mechanical mechanical (CMM, jointed arm)(CMM, jointed arm)
contactcontact ultrasonic trackersultrasonic trackersti t kti t k
inertial inertial (gyroscope, accelerometer)(gyroscope, accelerometer)
industrial CTindustrial CT
magnetic trackersmagnetic trackers
rangerangeacquisitionacquisition transmissivetransmissive
industrial CTindustrial CTultrasoundultrasoundMRIMRI
nonnon--opticaloptical radarradar
MRIMRI
reflectivereflectivenonnon opticaloptical
opticaloptical
sonarsonar
pp
![Page 5: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/5.jpg)
Range acquisition taxonomyshape from X:shape from X:
stereostereo
passivepassive
stereostereomotionmotionshadingshadingtexturetexture
ti lti l
passivepassive texturetexturefocusfocusdefocusdefocus
opticalopticalmethodsmethods active variants of passive methodsactive variants of passive methods
Stereo w. projected textureStereo w. projected texture
activeactive
p jp jActive depth from defocusActive depth from defocusPhotometric stereoPhotometric stereo
time of flighttime of flight
t i l tit i l titriangulationtriangulation
![Page 6: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/6.jpg)
Outline
• Passive approachesSt– Stereo
– Multiview approach
h• Active approaches– Triangulation– Shadow scanning
• Active variants of passive approachesp pp– Photometric stereo– Example-based photometric stereop p
![Page 7: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/7.jpg)
Passive approachesPassive approaches
![Page 8: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/8.jpg)
Stereo
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
![Page 9: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/9.jpg)
Stereo
• One distinguishable point being observedTh i b f d t th i t ti f – The preimage can be found at the intersection of the rays from the focal points to the image points
![Page 10: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/10.jpg)
Stereo
• Many points being observedN d th d t t bli h d– Need some method to establish correspondences
![Page 11: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/11.jpg)
Components of stereo vision systems
• Camera calibrationI ifi i i lifi h h f • Image rectification: simplifies the search for correspondences
• Correspondence: which item in the left image corresponds to which item in the right image
• Reconstruction: recovers 3-D information from the 2-D correspondencesp
![Page 12: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/12.jpg)
Epipolar geometry
• Epipolar constraint: corresponding points must lie on conjugate epipolar lineslie on conjugate epipolar lines– Search for correspondences becomes a 1-D problem
0'Τ F 0' =Τ Fxx
![Page 13: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/13.jpg)
Image rectification
Warp images such • Warp images such that conjugate epipolar lines become epipolar lines become collinear and parallel to u axisto u axis
![Page 14: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/14.jpg)
Disparity
• With rectified images, disparity is just (horizontal) displacement of corresponding (horizontal) displacement of corresponding features in the two images
Di it 0 f di t t i t– Disparity = 0 for distant points– Larger disparity for closer points
D th f i t ti l t 1/di it– Depth of point proportional to 1/disparity
![Page 15: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/15.jpg)
Reconstruction
• GeometricC t t th li t di l t R d R' – Construct the line segment perpendicular to R and R' that intersects both rays and take its mid-point
![Page 16: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/16.jpg)
Basic stereo algorithm
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
I t t h i dImprovement: match windows
![Page 17: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/17.jpg)
Basic stereo algorithm
• For each pixelF h di it– For each disparity
• For each pixel in window– Compute differenceCompute difference
– Find disparity with minimum SSD
![Page 18: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/18.jpg)
Reverse order of loops• For each disparity
F h i l– For each pixel• For each pixel in window
– Compute differenceCompute difference
• Find disparity with minimum SSD at each pixel
![Page 19: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/19.jpg)
Incremental computation
• Given SSD of a window, at some disparity
Image 1Image 1
Image 2Image 2Image 2Image 2
![Page 20: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/20.jpg)
Incremental computation
• Want: SSD at next location
Image 1Image 1
Image 2Image 2Image 2Image 2
![Page 21: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/21.jpg)
Incremental computation
• Subtract contributions from leftmost column, add contributions from rightmost columnadd contributions from rightmost column
++--
Image 1Image 1++++++++++
------
++--
I 2I 2----
++++
Image 2Image 2 ------
++++++
![Page 22: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/22.jpg)
Selecting window size
• Small window: more detail, but more noiseL i d b l d il• Large window: more robustness, less detail
• Example:
![Page 23: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/23.jpg)
Selecting window size
3 pixel window3 pixel window 20 pixel window20 pixel window
![Page 24: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/24.jpg)
Non-square windows
• Compromise: have a large window, but higher weight near the centerweight near the center
• Example: Gaussian• Example: Shifted windows
![Page 25: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/25.jpg)
Ordering constraint
• Order of matching features usually the samein both imagesin both images
• But not always: occlusion
![Page 26: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/26.jpg)
Dynamic programming
• Treat feature correspondence as graph problem
Right image featuresRight image features
11 22 33 4411 Cost of edges =Cost of edges =
Left imageLeft imagefeaturesfeatures
22
33
ggsimilarity ofsimilarity of
regions betweenregions betweenimage featuresimage featuresfeaturesfeatures 33
44
image featuresimage features
![Page 27: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/27.jpg)
Dynamic programming
• Find min-cost path through graph
Right image featuresRight image features
11 22 33 4411
11 11Left imageLeft imagefeaturesfeatures
22
33
11 11
222233featuresfeatures 33
443344
3344
![Page 28: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/28.jpg)
Energy minimization
• Another approach to improve quality of correspondencescorrespondences
• Assumption: disparities vary (mostly) smoothly• Minimize energy function:
Edata+λEsmoothnessdata smoothness
• Edata: how well does disparity match dataE : how well does disparity match• Esmoothness: how well does disparity matchthat of neighbors – regularization
![Page 29: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/29.jpg)
Energy minimization
• If data and energy terms are nice (continuous, smooth etc ) can try to minimize via gradient smooth, etc.) can try to minimize via gradient descent, etc.I ti di iti l i i th• In practice, disparities only piecewise smooth
• Design smoothness function that doesn’t penalize large jumps too much– Example: V(α,β)=min(|α−β|, K)
![Page 30: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/30.jpg)
Stereo as energy minimization• Matching Cost Formulated as Energy
“d t ” t li i b d t h– “data” term penalizing bad matches
),(),(),,( ydxyxdyxD +−= JI
– “neighborhood term” encouraging spatial smoothness
),(),(),,( yyy
similar)something(or
d2 and d1 labels with pixelsadjacentofcost21 ),(dd
ddV−=
=
similar)something(or21 dd −=
∑∑ + )()( ddVdDE ∑∑ +=)2,2(),1,1(
2,21,1),(
, ),(),,(yxyxneighbors
yxyxyx
yx ddVdyxDE
![Page 31: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/31.jpg)
Energy minimization
• Hard to find global minima of non-smooth functionsfunctions– Many local minima
P bl NP h d– Provably NP-hard
• Practical algorithms look for approximate l d lminima (e.g., simulated annealing)
![Page 32: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/32.jpg)
Stereo results– Data from University of Tsukuba
ground truthscene
htt // t iddl b d / t /http://cat.middlebury.edu/stereo/
![Page 33: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/33.jpg)
Results with window correlation
normalized correlation(best window size)
ground truth(best window size)
![Page 34: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/34.jpg)
Results with graph cuts
ground truthgraph cuts(Potts model E,(Potts model E,
expansion move algorithm)
![Page 35: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/35.jpg)
Stereo evaluation
![Page 36: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/36.jpg)
Stereo—best algorithms
![Page 37: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/37.jpg)
Volumetric multiview approaches
• Goal: find a model consistent with images“M d l i ” ( i i )• “Model-centric” (vs. image-centric)
• Typically use discretized volume (voxel grid)• For each voxel, compute occupied / free
(for some algorithms, also color, etc.)( g , , )
![Page 38: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/38.jpg)
Photo consistency
• Result: not necessarily the correct sceneM d h i• Many scenes produce the same images
True sceneTrue scene ReconstructedReconstructedscenescene
All scenesAll scenes
PhotoPhoto--consistent scenesconsistent scenes
![Page 39: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/39.jpg)
Silhouette carving
• Find silhouettes in all imagesE i• Exact version:– Back-project all silhouettes, find intersection
Binary ImagesBinary ImagesBinary ImagesBinary Images
![Page 40: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/40.jpg)
Silhouette carving
• Find silhouettes in all imagesE i• Exact version:– Back-project all silhouettes, find intersection
![Page 41: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/41.jpg)
Silhouette carving
• Limit of silhouette carving is visual hull orline hullline hull
• Complement of lines that don’t intersect object• In general not the same as object
– Can’t recover “pits” in object
• Not the same as convex hull
![Page 42: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/42.jpg)
Silhouette carving
• Discrete version:L ll l i l– Loop over all voxels in some volume
– If projection into images lies inside all silhouettes, mark as occupiedmark as occupied
– Else mark as free
![Page 43: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/43.jpg)
Silhouette carving
![Page 44: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/44.jpg)
Voxel coloring
• Seitz and Dyer, 1997I ddi i f / i d l• In addition to free / occupied, store colorat each voxel
• Explicitly accounts for occlusion
![Page 45: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/45.jpg)
Voxel coloring
• Basic idea: sweep through a voxel gridP j t h l i t h i i hi h– Project each voxel into each image in whichit is visibleIf colors in images agree mark voxel with color– If colors in images agree, mark voxel with color
– Else, mark voxel as empty
A t f l b d i • Agreement of colors based on comparing standard deviation of colors to threshold
![Page 46: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/46.jpg)
Voxel coloring and occlusion
![Page 47: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/47.jpg)
Voxel coloring and occlusion
• Problem: which voxels are visible?S l i i i• Solution: constrain camera views– When a voxel is considered, necessary occlusion
i f ti t b il blinformation must be available– Sweep occluders before occludees
C ll h – Constrain camera positions to allow this sweep
![Page 48: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/48.jpg)
Voxel coloring sweep order
SceneSceneTraversalTraversal
![Page 49: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/49.jpg)
Voxel coloring camera positions
Inward-looking Outward-lookinga d oo gCameras above scene
Out a d oo gCameras inside scene
SeitzSeitz
![Page 50: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/50.jpg)
Image acquisition
Selected Dinosaur ImagesSelected Dinosaur Images
•Calibrated Turntable•360° rotation (21 images)
S l t d Fl IS l t d Fl ISelected Flower ImagesSelected Flower Images
![Page 51: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/51.jpg)
Voxel coloring results
Dinosaur ReconstructionDinosaur Reconstruction72 K l l d72 K l l d
Flower ReconstructionFlower Reconstruction70 K l l d70 K l l d72 K voxels colored72 K voxels colored
7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGIon a 250MHz SGIon a 250MHz SGI on a 250MHz SGIon a 250MHz SGI
![Page 52: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/52.jpg)
Space carving
Image 1Image 1 Image NImage NImage 1Image 1 Image NImage N
…...…...
Initialize to a volume V containing the true sceneInitialize to a volume V containing the true sceneChoose a voxel on the current surfaceChoose a voxel on the current surface
R t tilR t tilCarve if not photoCarve if not photo--consistentconsistentProject to visible input imagesProject to visible input images
Repeat until convergenceRepeat until convergence
![Page 53: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/53.jpg)
Multi-pass plane sweep
• Faster alternative:S l i h f 6 i i l di ti– Sweep plane in each of 6 principal directions
– Consider cameras on only one side of planeR t til – Repeat until convergence
![Page 54: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/54.jpg)
Multi-pass plane sweep
True Scene Reconstruction
![Page 55: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/55.jpg)
Multi-pass plane sweep
![Page 56: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/56.jpg)
Multi-pass plane sweep
![Page 57: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/57.jpg)
Multi-pass plane sweep
![Page 58: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/58.jpg)
Multi-pass plane sweep
![Page 59: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/59.jpg)
Space carving results: African violet
Input image (1 of 45) Input image (1 of 45) ReconstructionReconstruction
ReconstructionReconstructionReconstructionReconstruction
![Page 60: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/60.jpg)
Space carving results: hand
Input imageInput image(1 of 100) (1 of 100)
ReconstructionReconstructionReconstructionReconstruction
![Page 61: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/61.jpg)
Active approachesActive approaches
![Page 62: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/62.jpg)
Time of flight
• Basic idea: send out pulse of light (usually laser) time how long it takes to returnlaser), time how long it takes to return
tcr Δ=212
![Page 63: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/63.jpg)
Laser scanning (triangulation)
Object
Direction of travel
Laser sheet
CCD image plane
CCDLaserCylindrical lens
Digital Michelangelo Projecthttp://graphics stanford edu/projects/mich/
• Optical triangulation– Project a single stripe of laser light
http://graphics.stanford.edu/projects/mich/
– Scan it across the surface of the object– This is a very precise version of structured light scanning
Other patterns are possible• Other patterns are possible
![Page 64: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/64.jpg)
Cyberware
face and hand full body
![Page 65: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/65.jpg)
XYZRGB
![Page 66: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/66.jpg)
XYZRGB
![Page 67: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/67.jpg)
Shadow scanning
DeskL Stick orLamp pencil
CameraDesk
http://www.vision.caltech.edu/bouguetj/ICCV98/
![Page 68: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/68.jpg)
Basic idea
• Calibration issues:– where’s the camera wrt. ground plane?– where’s the shadow plane?
• depends on light source position, shadow edge
![Page 69: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/69.jpg)
Two Plane Version
• Advantagesg– don’t need to pre-calibrate the light source– shadow plane determined from two shadow edgesshadow plane determined from two shadow edges
![Page 70: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/70.jpg)
Estimating shadow lines
![Page 71: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/71.jpg)
Shadow scanning in action
![Page 72: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/72.jpg)
Results
accuracy: 0.1mm over 10cm ~ 0.1% error
![Page 73: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/73.jpg)
Textured objects
![Page 74: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/74.jpg)
Scanning with the sun
accuracy: 1mm over 50cm ~ 0.5% error
![Page 75: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/75.jpg)
Scanning with the sun
accuracy: 1cm over 2maccuracy: 1cm over 2m~ 0.5% error
![Page 76: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/76.jpg)
Active variants of passive approaches
![Page 77: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/77.jpg)
The BRDF• The Bidirectional Reflection Distribution Function
– Given an incoming ray and outgoing rayg y g g ywhat proportion of the incoming light is reflected along outgoing ray?
surface normalsurface normal
γρ )(),( nlvlI ⋅×=
![Page 78: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/78.jpg)
Diffuse reflection (Lambertian)
��
�
�
dkvl =),(ρ albedo
Assuming that light strength is 1.g g g .
![Page 79: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/79.jpg)
Photometric stereo
N
L1
L2
V
L3
• Can write this as a matrix equation:• Can write this as a matrix equation:
![Page 80: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/80.jpg)
Solving the equations
![Page 81: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/81.jpg)
More than three lights
• Get better results by using more lights
L l i• Least squares solution:
• Solve for N, kd as before, d
![Page 82: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/82.jpg)
Trick for handling shadows
• Weight each equation by the pixel brightness:
• Gives weighted least-squares matrix g qequation:
• Solve for N, kd as before, d
![Page 83: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/83.jpg)
Photometric Stereo Setup
![Page 84: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/84.jpg)
Procedure
• Calibrate cameraC lib li h di i /i i i• Calibrate light directions/intensities
• Photographing objects (HDR recommended)• Estimate normals• Estimate depth• Estimate depth
![Page 85: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/85.jpg)
Estimating light directions
• Trick: place a chrome sphere in the scene
– the location of the highlight tells you where the light source issource is
• Use a ruler
![Page 86: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/86.jpg)
Photographing objects
![Page 87: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/87.jpg)
Normalize light intensities
![Page 88: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/88.jpg)
Estimate normals
![Page 89: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/89.jpg)
Depth from normals
![Page 90: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/90.jpg)
Results
![Page 91: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/91.jpg)
Limitations
• Big problemsd ’t k f hi thi i t l t – doesn’t work for shiny things, semi-translucent thingsshadows inter reflections– shadows, inter-reflections
• Smaller problemsl b – calibration requirements
• measure light source directions, intensities• camera response function• camera response function
![Page 92: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/92.jpg)
Example-based photometric stereo
• Estimate 3D shape by varying illumination, fixed camerafixed camera
• Operating conditions– any opaque material– distant camera, lighting– reference object available– no shadows, interreflections, transparency
![Page 93: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/93.jpg)
same surface normal
“Orientation consistency”
![Page 94: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/94.jpg)
![Page 95: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/95.jpg)
![Page 96: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/96.jpg)
![Page 97: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/97.jpg)
![Page 98: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/98.jpg)
![Page 99: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/99.jpg)
![Page 100: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/100.jpg)
Virtual views
![Page 101: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/101.jpg)
Velvet
![Page 102: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/102.jpg)
Virtual Views
![Page 103: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/103.jpg)
Brushed Fur
![Page 104: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/104.jpg)
Virtual Views
![Page 105: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/105.jpg)
![Page 106: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/106.jpg)
Active stereo with structured light
Li Zhang’s one-shot stereo
camera 1 camera 1
Li Zhang s one shot stereo
projector projector
• Project “structured” light patterns onto the objectcamera 2
– simplifies the correspondence problem
![Page 107: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/107.jpg)
Spacetime Stereo
http://grail.cs.washington.edu/projects/stfaces/
![Page 108: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/108.jpg)
3D Model Acquisition Pipeline
3D Scanner3D Scanner
![Page 109: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/109.jpg)
3D Model Acquisition Pipeline
3D Scanner3D Scanner
View PlanningView PlanningView PlanningView Planning
![Page 110: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/110.jpg)
3D Model Acquisition Pipeline
3D Scanner3D Scanner
AlignmentAlignmentView PlanningView Planning AlignmentAlignmentView PlanningView Planning
![Page 111: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/111.jpg)
3D Model Acquisition Pipeline
3D Scanner3D Scanner
AlignmentAlignmentViewView PlanningPlanning AlignmentAlignmentView View PlanningPlanning
MergingMerging
![Page 112: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/112.jpg)
Volumetric reconstruction
![Page 113: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/113.jpg)
Signed distance function
![Page 114: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/114.jpg)
Results
![Page 115: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/115.jpg)
The Digital Michelangelo Project
• Goal: scan 10 sculptures by MichelangeloHi h l i (“ illi ”) • High-resolution (“quarter-millimeter”) geometry
• Stanford University, led by Marc Levoy
![Page 116: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/116.jpg)
Systems, projects and applicationsSystems, projects and applications
![Page 117: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/117.jpg)
Scanning the David
height of gantry: 7.5 metersi h f 800 kilweight of gantry: 800 kilograms
![Page 118: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/118.jpg)
Range processing pipeline
• steps1. manual initial alignment2. ICP to one existing scan3. automatic ICP of all overlapping pairs4. global relaxation to spread out error5 merging using volumetric method5. merging using volumetric method
![Page 119: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/119.jpg)
Statistics about the scan
• 480 individually aimed scans• 2 billion polygons• 7,000 color images• 32 gigabytesg g y• 30 nights of scanning• 22 peoplep p
![Page 120: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/120.jpg)
Comparison
photograph 1 0 mm computer modelphotograph 1.0 mm computer model
![Page 121: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/121.jpg)
Results
![Page 122: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/122.jpg)
The Great Buddha Project
• Great Buddha of KamakuraO i i l d f d l d 1243• Original made of wood, completed 1243
• Covered in bronze and gold leaf, 1267• Approx. 15 m tall• Goal: preservation of• Goal: preservation of
cultural heritage• Institute of Industrial Science• Institute of Industrial Science,
University of Tokyo, led byK t hi Ik hiK t hi Ik hiKatsushi IkeuchiKatsushi Ikeuchi
![Page 123: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/123.jpg)
Scanner• Cyrax range scanner
by Cyra Technologiesby Cyra Technologies• Laser pulse time-of-flight• Accuracy: 4 mm• Range: 100 mg
![Page 124: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/124.jpg)
Processing• 20 range images (a few million points)
Si l ll ll ICP• Simultaneous all-to-all ICP• Variant of volumetric merging (parallelized)
![Page 125: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/125.jpg)
Results
![Page 126: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/126.jpg)
Applications in VFX
• 3D scanningH b id f IMAX• Hybrid camera for IMAX
• View interpolation
![Page 127: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/127.jpg)
3D scanning
XYZRGB IncXYZRGB Inc.
![Page 128: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/128.jpg)
IMAX 3D
• 6K resolution, 42 linear bits per pixelF CG i i ll k 6 h f f• For CG, it typically takes 6 hours for a frame
• 45-minute IMAX 3D CG film requires a 100-CPU rendering farm full-time for about a year just for rendering
• For live-action, camera is bulky (like a refrigerator)g )
![Page 129: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/129.jpg)
Hybrid stereo camera
![Page 130: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/130.jpg)
Live-action sequence
![Page 131: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/131.jpg)
Hybrid input
left
right
![Page 132: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/132.jpg)
Hybrid input
left
left
right
![Page 133: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/133.jpg)
Combine multiple hires to lores
![Page 134: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/134.jpg)
Results
![Page 135: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/135.jpg)
View interpolation
Bullet time videoBullet time video
![Page 136: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/136.jpg)
View interpolation
High quality video view interpolationHigh-quality video view interpolation
![Page 137: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/137.jpg)
Final projectFinal project
![Page 138: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/138.jpg)
Final project
• Assigned: 5/14D ?• Due: ?
• Proposal by 5/27
![Page 139: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/139.jpg)
Final project
• Research (1-2 people)S (1 3 l )• System (1-3 people)
• Film (3-4 people)
![Page 140: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/140.jpg)
Research
• Define a problem and try to solve itY d ’ d l i ll b h k • You don’t need to solve it all, but have to make a reasonable progress, for example, solve a i lifi d isimplified version.
• Find inspirations from SIGGRAPH/CVPR/ICCV papers
![Page 141: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/141.jpg)
System
• Implement existing algorithm into a useful system such as implementing recent SIGGRAPH system such as implementing recent SIGGRAPH or CVPR/ICCV papers
![Page 142: lec12 3dphoto.ppt [相容模式] - 國立臺灣大學cyy/courses/vfx/09spring/lectures/... · • I ifi i i lifi h h f Image rectification: simplifies the search for correspondences](https://reader030.fdocuments.in/reader030/viewer/2022040600/5e89e8842819ef3be6509875/html5/thumbnails/142.jpg)
Film
• It must be an “effect” film. Y l B I • You can use any tools as you want. But, I assume that you have to write some on your own.
• Find inspirations fromGatech’s vfx course http://www.cc.gatech.edu/classes/AY2004/cs4480_spring/independent film makersindependent film makershttp://www.peerlessproductions.com/ADs/films/YouTube
• Submit two videos, final and making-of.