Computational Photography - B3...
Transcript of Computational Photography - B3...
![Page 1: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/1.jpg)
![Page 2: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/2.jpg)
Paul Paul DebevecDebevec
Computational Photography:Advanced Topics
Computational Photography:Computational Photography:Advanced TopicsAdvanced Topics
![Page 3: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/3.jpg)
Class: Class: Computational Photography, Advanced TopicsComputational Photography, Advanced TopicsDebevecDebevec, Raskar and , Raskar and TumblinTumblin
Module 1: 105 minutesModule 1: 105 minutes
1:45: A.1 Introduction and Overview 1:45: A.1 Introduction and Overview (Raskar, 15 minutes)(Raskar, 15 minutes)
2:00: A.2 Concepts in Computational Photography 2:00: A.2 Concepts in Computational Photography ((TumblinTumblin, 15 minutes) , 15 minutes)
2:15: A.3 Optics: Computable Extensions 2:15: A.3 Optics: Computable Extensions (Raskar, 30 minutes) (Raskar, 30 minutes)
2:45: A.4 Sensor Innovations2:45: A.4 Sensor Innovations ((TumblinTumblin, 30 minutes), 30 minutes)
3:15: Q & A3:15: Q & A (15 minutes)(15 minutes)
3:30: Break: 15 minutes3:30: Break: 15 minutes
Module 2: 105 minutesModule 2: 105 minutes
3:45: B.1 Illumination As Computing3:45: B.1 Illumination As Computing ((DebevecDebevec, 25 minutes) , 25 minutes)
4:10: B.2 Scene and Performance Capture4:10: B.2 Scene and Performance Capture ((DebevecDebevec, 20 minutes), 20 minutes)
4:30: B.3 Image Aggregation & Sensible Extensions4:30: B.3 Image Aggregation & Sensible Extensions ((TumblinTumblin, 20 minutes), 20 minutes)
4:50: B.4 Community and Social Impact 4:50: B.4 Community and Social Impact (Raskar, 20 minutes)(Raskar, 20 minutes)
5:10: B.4 Panel discussion 5:10: B.4 Panel discussion (All, 20 minutes) (All, 20 minutes)
Class Page : Class Page : http://http://ComputationalPhotography.orgComputationalPhotography.org
![Page 4: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/4.jpg)
B3:B3: Image Aggregation Image Aggregation and Sensible Extensionsand Sensible Extensions
(20 minutes)(20 minutes)
Computational Photography: Advanced Topics
Jack TumblinJack TumblinNorthwestern UniversityNorthwestern University
![Page 5: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/5.jpg)
Image AssemblyImage AssemblyImage Assembly
•• Unordered images toUnordered images to……
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 6: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/6.jpg)
Image AssemblyImage AssemblyImage Assembly
•• Unordered images toUnordered images to……
Aligned subset toAligned subset to……
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 7: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/7.jpg)
Image AssemblyImage AssemblyImage Assembly
•• Unordered images toUnordered images to……
Fully Aligned toFully Aligned to……
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 8: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/8.jpg)
Image AssemblyImage AssemblyImage Assembly
•• Unordered images toUnordered images to……
Fully Aligned & matched.Fully Aligned & matched.
•• Panoramas: thousands on Panoramas: thousands on FlickrFlickr ‘‘autostitchautostitch’’http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 9: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/9.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
NEARNEAR
![Page 10: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/10.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
FARFAR
![Page 11: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/11.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 12: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/12.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 13: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/13.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 14: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/14.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 15: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/15.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 16: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/16.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 17: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/17.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 18: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/18.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 19: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/19.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 20: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/20.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 21: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/21.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 22: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/22.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 23: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/23.jpg)
FUSION: BestFUSION: Best--Focus DistanceFocus Distance
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 24: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/24.jpg)
Source images
FUSION ‘Graph Cuts’ Solution
Agrawala et al., Digital PhotomontageSIGGRAPH 2004
![Page 25: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/25.jpg)
High Speed VideoHigh Speed Video
Say you want 120 frame per second (fps) video.• You could get one camera that runs at 120 fps• Or…
![Page 26: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/26.jpg)
High Speed VideoHigh Speed Video
Say you want 120 frame per second (fps) video. • You could get one camera that runs at 120 fps• Or… get 4 cameras running at 30 fps.
![Page 27: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/27.jpg)
52 Camera Cluster, 1560 FPS52 Camera Cluster, 1560 FPSLevoy et al., SIGGRAPH2005
![Page 28: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/28.jpg)
What else can we extend? What else can we extend? FilmFilm--Like Camera Parameters: Like Camera Parameters: •• Field of View: image stitching for panoramasField of View: image stitching for panoramas•• Dynamic Range: Dynamic Range: Radiance Maps Radiance Maps •• Frame Rate: Interleaved VideoFrame Rate: Interleaved Video•• Resolution: Resolution: ‘‘SuperSuper--resolutionresolution’’ methods methods
Visual Appearance & Content:Visual Appearance & Content:
![Page 29: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/29.jpg)
What else can we extend? What else can we extend? FilmFilm--Like Camera Parameters: Like Camera Parameters:
•• Field of View: image stitching for panoramasField of View: image stitching for panoramas
•• Dynamic Range: Dynamic Range: Radiance MapsRadiance Maps
•• Frame Rate: Interleaved VideoFrame Rate: Interleaved Video [Levoy][Levoy]
•• Resolution: Resolution: ‘‘SuperSuper--resolutionresolution’’ methods methods [Nayar][Nayar]
Visual Appearance & Available Content:Visual Appearance & Available Content:
•• Temporal Continuity: SpaceTemporal Continuity: Space--time mixturestime mixtures
•• Viewpoint Constraints: PhotoViewpoint Constraints: Photo--Tourism, flexible collage Tourism, flexible collage
•• Seam Carving & Resizing: leastSeam Carving & Resizing: least--loss deletions loss deletions
•• Learning from Large Datasets: day/night, scene elementsLearning from Large Datasets: day/night, scene elements
•• Transfer Visible Details Transfer Visible Details –– MultiMulti--Light, IR relightingLight, IR relighting……
------ DONE ! DONE ! ------ (mostly)
(mostly)
![Page 30: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/30.jpg)
Visually Expressive Visually Expressive ‘‘Time FragmentsTime Fragments’’
•• DuchampDuchamp–– Nude Descending a StaircaseNude Descending a Staircase
![Page 31: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/31.jpg)
FUSION: Time/Space MixturesFUSION: Time/Space Mixtures
Richard Hundley 2001Richard Hundley 2001
![Page 32: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/32.jpg)
Strip Photography (1960s, Strip Photography (1960s, DavidhazyDavidhazy))
•• Time increases leftTime increases left--toto--right;right;
•• Girl in rotating chairGirl in rotating chair
![Page 33: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/33.jpg)
![Page 34: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/34.jpg)
TimeTime--Space FUSION:Space FUSION:ShapeShape--Time PhotographyTime Photography
Freeman et al 2003
![Page 35: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/35.jpg)
Webcam Image sets:Webcam Image sets:What can we learn?What can we learn?
AArchive of rchive of MMany any OOutdoor utdoor SScenes (AMOS)cenes (AMOS)
•• 2006 start 2006 start
•• >500 webcams>500 webcams
•• >20 million photos >20 million photos
http://www.cse.wustl.edu/~jacobsn/projects/webcam_dataset/
Hour0 2412
Jan
June
DecDate
![Page 36: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/36.jpg)
Temporal Variations: VERY consistentTemporal Variations: VERY consistent
PCA on image setsPCA on image sets
•• 11stst 4 components:4 components:hourhour--ofof--dayday
•• 22ndnd Order statisticsOrder statistics
Annotate images for:Annotate images for:
•• Weather, Seasons,Weather, Seasons,
•• Surface Surface normalsnormals,,
•• Geog. OrientationsGeog. Orientations……
![Page 37: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/37.jpg)
Temporal Variations: VERY consistentTemporal Variations: VERY consistent
From timeFrom time--stamped stamped
image sequenceimage sequence
Find GeographicFind Geographic
Location within Location within
50 miles50 miles
Jacobs, Pless et al; “Geolocating static cameras” (ICCV 20
![Page 38: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/38.jpg)
Photo TourismPhoto Tourism
•• Many Many uncalibrateduncalibrated photos of popular sitesphotos of popular sites
•• Each have many 3D scene pointEach have many 3D scene point--correspondencescorrespondences
•• Find, solve for 3D scene points & each cameraFind, solve for 3D scene points & each camera’’s 3D poses 3D pose
•• ReprojectReproject on 3D shapes, use view interpolationon 3D shapes, use view interpolation
•• Friendly UI sketchFriendly UI sketch--like: explore by 3D flyinglike: explore by 3D flying……
““Photo Tourism: Exploring Photo collections In in 3DPhoto Tourism: Exploring Photo collections In in 3D”” SnavelySnavely et al., SIGGRAPH 2006et al., SIGGRAPH 2006
TRY IT! http://phototour.cs.washington.edu/
![Page 39: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/39.jpg)
Feature matchingFeature matching““Photo Tourism: Exploring Photo collections In in 3DPhoto Tourism: Exploring Photo collections In in 3D”” SnavelySnavely et al., SIGGRAPH 2006et al., SIGGRAPH 2006
•• Find Features [SIFT, Lowe IJCV04], thenFind Features [SIFT, Lowe IJCV04], then
•• Find Find pairwisepairwise matches (refined by RANSAC), thenmatches (refined by RANSAC), then
•• Find connected pairsFind connected pairs’’ components, then components, then SfMSfM, , ……
Image 1Image 3
Image 2 Image 4
![Page 40: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/40.jpg)
‘‘Structure Structure from Motionfrom Motion’’
Camera 1
Camera 2
Camera 3R1,T1
R2,T2
R3,T3
p1
p4
p3
p2
p5
p6
p7
minimizef (R,T,P)
Photo TourismPhoto Tourism““Photo Tourism: Exploring Photo collections In in 3DPhoto Tourism: Exploring Photo collections In in 3D”” SnavelySnavely et al., SIGGRAPH 2006et al., SIGGRAPH 2006
![Page 41: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/41.jpg)
Photo TourismPhoto Tourism
•• sdsd““Photo Tourism: Exploring Photo collections In in 3DPhoto Tourism: Exploring Photo collections In in 3D”” SnavelySnavely et al., SIGGRAPH 2006et al., SIGGRAPH 2006
![Page 42: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/42.jpg)
A tiled camera arrayA tiled camera array
•• 12 12 ×× 8 array of VGA cameras8 array of VGA cameras
•• abutted: abutted: 7680 7680 ×× 3840 pixels3840 pixels
•• overlapped 50%: half of thisoverlapped 50%: half of this
•• total field of view = 29total field of view = 29°° widewide
•• (seamless mosaic: not hard) (seamless mosaic: not hard)
•• cameras individually meteredcameras individually metered
•• Approx same centerApprox same center--ofof--projproj..
“High performance imaging using large camera arrays”Wilburn, Levoy et al.; SIGGRAPH 2005)
![Page 43: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/43.jpg)
Tiled panoramic image (before)Tiled panoramic image (before)
“High performance imaging using large camera arrays” Wilburn, Levoy SIGG’05)
![Page 44: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/44.jpg)
Tiled panoramic image (after)Tiled panoramic image (after)
“High performance imaging using large camera arrays” Wilburn, Levoy SIGG’05)
![Page 45: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/45.jpg)
Scene Collage & Flexible Camera ArraysScene Collage & Flexible Camera Arrays"Scene Collages and Flexible Camera Arrays," Y. Nomura, L. Zhang and S.K. Nayar, EGSR 2007.
•• bend the camerabend the camera--sheet to fit the scenesheet to fit the scene
•• Align, mix, blend photos in video collageAlign, mix, blend photos in video collage
![Page 46: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/46.jpg)
Scene Collage & Flexible Camera ArraysScene Collage & Flexible Camera Arrays
..
"Scene Collages and Flexible Camera Arrays," Y. Nomura, L. Zhang and S.K. Nayar, EGSR 2007.
![Page 47: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/47.jpg)
MultiMulti--Light Detail TransferLight Detail Transfer
•• Different light Different light Different visible detailsDifferent visible details
•• Extract, Control/Enhance, Merge detailsExtract, Control/Enhance, Merge details
SIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections
![Page 48: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/48.jpg)
MultiMulti--Light Detail TransferLight Detail TransferSIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections
•• Different light Different light Different visible detailsDifferent visible details
•• Extract, Control/Enhance, Merge detailsExtract, Control/Enhance, Merge details
Light 1
![Page 49: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/49.jpg)
MultiMulti--Light Detail TransferLight Detail TransferSIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections
•• Different light Different light Different visible detailsDifferent visible details
•• Extract, Control/Enhance, Merge detailsExtract, Control/Enhance, Merge details
Light 2
![Page 50: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/50.jpg)
MultiMulti--Light Detail TransferLight Detail TransferSIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections
•• Different light Different light Different visible detailsDifferent visible details
•• Extract, Control/Enhance, Merge detailsExtract, Control/Enhance, Merge detailsLight 3
![Page 51: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/51.jpg)
MultiMulti--Light Detail TransferLight Detail TransferSIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections
•• Different light Different light Different visible detailsDifferent visible details
•• Extract, Control/Enhance, Merge detailsExtract, Control/Enhance, Merge details
•• Bilateral filtersBilateral filters
•• UserUser--set weightsset weights
•• Adjust to suitAdjust to suit……flat, detailed orflat, detailed orwith shadowswith shadows
![Page 52: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/52.jpg)
MultiMulti--Light Detail TransferLight Detail TransferSIGG2007 Fattal et al., Multiscale Shape and Detail Enhancement from Multi-light Image Collections
•• Different light Different light Different visible detailsDifferent visible details
•• Extract, Control/Enhance, Merge detailsExtract, Control/Enhance, Merge details
•• Bilateral filtersBilateral filters
•• UserUser--set weightsset weights
•• Adjust to suitAdjust to suit……flat, detailed orflat, detailed orwith shadowswith shadows
![Page 53: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/53.jpg)
FUSION: Multispectral WavelengthsFUSION: Multispectral Wavelengths
Vegetation Mapping of the Forest
=+
SAR Optical Landsat
![Page 54: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/54.jpg)
FUSION: Multispectral WavelengthsFUSION: Multispectral Wavelengths
Vegetation Mapping of the Forest
SAR Optical Landsat
=+
![Page 55: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/55.jpg)
Bennett2007: Multispectral Video FusionBennett2007: Multispectral Video Fusion
DualDual--Bilateral filter: Bilateral filter:
fuses best of visible + IRfuses best of visible + IR
![Page 56: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/56.jpg)
Video Relighting from IR illuminationVideo Relighting from IR illuminationEG2008, EG2008, Wang,DavisWang,Davis et al.et al.”” Video Relighting Using Infrared IlluminationVideo Relighting Using Infrared Illumination””..
![Page 57: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/57.jpg)
Video Relighting from IR IlluminationVideo Relighting from IR Illumination
•• Switched IR illuminators, 8 photos per frameSwitched IR illuminators, 8 photos per frame
•• Ratio ImagesRatio Images
•• Hue CorrectionsHue Corrections
![Page 58: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/58.jpg)
Seam Carving: ContentSeam Carving: Content--Aware ResizingAware Resizing
•• Find Find ‘‘leastleast--damagingdamaging’’ seamseam( ( toptop bottombottom or or leftleft rightright connected pathconnected path))
•• Remove seam, or insert a similar oneRemove seam, or insert a similar one……
SIGG2007: “Seam Carving for Content Aware Image Resizing” Avidan, ShamirSEE ALSO SIGG 2008: “Improved Seam Carving for Video Retargeting” Rubenstien et al.
![Page 59: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/59.jpg)
Seam Carving: ContentSeam Carving: Content--Aware ResizingAware Resizing
•• Find Find ‘‘leastleast--damagingdamaging’’ seamseam( ( toptop bottombottom or or leftleft rightright connected pathconnected path))
•• Remove seam, or insert a similar oneRemove seam, or insert a similar one……
SIGG2007: “Seam Carving for Content Aware Image Resizing” Avidan, ShamirSEE ALSO SIGG 2008: “Improved Seam Carving for Video Retargeting” Rubenstien et al.
![Page 60: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/60.jpg)
Seam Carving: ContentSeam Carving: Content--Aware ResizingAware ResizingSIGG2007: “Seam Carving for Content Aware Image Resizing” Avidan, Shamir
•• Find, remove Find, remove ‘‘leastleast--damagingdamaging’’ seamseam( ( toptop bottombottom or or leftleft rightright connected pathconnected path))
SEE ALSO SIGG 2008: “Improved Seam Carving for Video Retargeting” Rubenstien et al.
![Page 61: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/61.jpg)
Perspective? Or Not?Perspective? Or Not?
Agrawala et al, “Long Scene Panoramas”, Siggraph 2006
Rademacher et al, MCOP, Siggraph 1998
![Page 62: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/62.jpg)
ConclusionsConclusions•• Merged Multiple Measurements:Merged Multiple Measurements:
MultiMulti--camera, multicamera, multi--sensor, sensor, multimulti--optics, multioptics, multi--lightinglighting
•• Can escape obvious Optical Limits, sureCan escape obvious Optical Limits, sure……–– lens diffraction limits, noise, available lightlens diffraction limits, noise, available light
–– Measurement time, position, occlusion, poseMeasurement time, position, occlusion, pose
•• But ALSO But ALSO ‘‘optically impossibleoptically impossible’’ visual experiences visual experiences –– Sunless shadows, Invisible lighting, wrapSunless shadows, Invisible lighting, wrap--around viewpoints around viewpoints
![Page 63: Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e717193513d692ad8029c32/html5/thumbnails/63.jpg)