Image-based rendering
description
Transcript of Image-based rendering
![Page 1: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/1.jpg)
Image-based rendering
Michael F. Cohen
Microsoft Research
![Page 2: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/2.jpg)
Computer Graphics
Image
Output
ModelSyntheticCamera
![Page 3: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/3.jpg)
Real Scene
Computer Vision
Real Cameras
Model
Output
![Page 4: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/4.jpg)
Combined
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
![Page 5: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/5.jpg)
But, vision technology falls short
ModelReal Scene
Real Cameras
Image
Output
SyntheticCamera
![Page 6: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/6.jpg)
… and so does graphics.
ModelReal Scene
Real Cameras
Image
Output
SyntheticCamera
![Page 7: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/7.jpg)
Image Based Rendering
Real Scene
Real Cameras-or-
Expensive Image Synthesis
Images+Model
Image
Output
SyntheticCamera
![Page 8: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/8.jpg)
Ray
Constant radiance• time is fixed
5D• 3D position• 2D direction
![Page 9: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/9.jpg)
All Rays
Plenoptic Function• all possible images• too much stuff!
![Page 10: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/10.jpg)
Line
Infinite line
4D• 2D direction• 2D position
![Page 11: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/11.jpg)
Ray
Discretize
Distance between 2 rays• Which is closer together?
![Page 12: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/12.jpg)
Image
What is an image?
All rays through a point• Panorama?
![Page 13: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/13.jpg)
Image
2D • position of rays has been fixed• direction remains
![Page 14: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/14.jpg)
Image
Image plane
2D• position
![Page 15: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/15.jpg)
Image plane
2D• position
Image
![Page 16: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/16.jpg)
Light leaving towards “eye”
2D• just dual of image
Object
![Page 17: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/17.jpg)
Object
All light leaving object
![Page 18: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/18.jpg)
Object
4D• 2D position• 2D direction
![Page 19: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/19.jpg)
Object
All images
![Page 20: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/20.jpg)
Lumigraph
How to • organize• capture• render
![Page 21: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/21.jpg)
Lumigraph - Organization
2D position 2D direction
s
![Page 22: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/22.jpg)
Lumigraph - Organization
2D position 2D position
2 plane parameterization
su
![Page 23: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/23.jpg)
Lumigraph - Organization
2D position 2D position
2 plane parameterization us
t s,tu,v
v
s,t
u,v
![Page 24: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/24.jpg)
Lumigraph - Organization
Hold s,t constant Let u,v vary An image
s,t u,v
![Page 25: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/25.jpg)
Lumigraph - Organization
Discretization• higher res near object
• if diffuse• captures texture
• lower res away• captures directions
s,t u,v
![Page 26: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/26.jpg)
Lumigraph - Capture
Idea 1• Move camera carefully
over s,t plane• Gantry
• see Lightfield paper
s,t u,v
![Page 27: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/27.jpg)
Lumigraph - Capture
Idea 2• Move camera anywhere• Rebinning
• see Lumigraph paper
s,t u,v
![Page 28: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/28.jpg)
Lumigraph - Rendering
For each output pixel• determine s,t,u,v• either
• find closest discrete RGB• interpolate near values
s,t u,v
![Page 29: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/29.jpg)
Lumigraph - Rendering
For each output pixel• determine s,t,u,v
• either• use closest discrete RGB• interpolate near values s u
![Page 30: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/30.jpg)
Lumigraph - Rendering
Nearest• closest s• closest u• draw it
Blend 16 nearest• quadrilinear interpolation s u
![Page 31: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/31.jpg)
High-Quality Video View Interpolation
Using a Layered Representation
Larry ZitnickSing Bing KangMatt UyttendaeleSimon WinderRick Szeliski
Interactive Visual Media GroupMicrosoft Research
![Page 32: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/32.jpg)
Current practice
Many cameras
Motion Jitter
vs.
free viewpoint video
![Page 33: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/33.jpg)
Current practice
Many cameras
Motion Jitter
vs.
free viewpoint video
![Page 34: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/34.jpg)
Video view interpolation
Fewer cameras
Smooth Motion
Automatic
and
Real-time rendering
![Page 35: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/35.jpg)
Prior work: IBR (static)
The LumigraphGortler et al., SIGGRAPH ‘96
Concentric MosaicsShum & He, SIGGRAPH ‘99
Plenoptic ModelingMcMillan & Bishop, SIGGRAPH ‘95
Light Field RenderingLevoy & Hanrahan, SIGGRAPH ‘96
![Page 36: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/36.jpg)
Prior work: IBR (dynamic)
Free-viewpoint Video of HumansCarranza et al., SIGGRAPH ‘03
Image-Based Visual HullsMatusik et al., SIGGRAPH ‘00
Virtualized RealityTM
Kanade et al., IEEE Multimedia ‘97Dynamic Light Fields
Goldlucke et al., VMV ‘02
Stanford Multi-Camera Array Project
3D TVMatusik & Pfister,
SIGGRAPH ‘04
![Page 37: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/37.jpg)
System overview
OFFLINE
ONLINE
Video Capture
Stereo Compression
SelectiveDecompression
Render
File
Representation
Video Capture
![Page 38: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/38.jpg)
concentratorsconcentrators
hard diskshard disks controlling
laptopcontrollinglaptop
camerascamerascameras
![Page 39: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/39.jpg)
Calibration
Zhengyou Zhang, 2000
![Page 40: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/40.jpg)
Input videos
![Page 41: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/41.jpg)
Key to view interpolation: Geometry
Stereo Geometry
Camera 1 Camera 2
Image 1 Image 2
Virtual Camera
![Page 42: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/42.jpg)
Match ScoreMatch ScoreMatch Score Match Score
Good
Bad
Image correspondence
Correct
Image 1 Image 2
Leg
Wall
Incorrect
![Page 43: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/43.jpg)
Image 1 Image 2
Local matching
Low textureLow texture
![Page 44: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/44.jpg)
Number of states = number of depth levels
Image 2Image 1
Global regularization
Create MRF (Markov Random Field):
A F
E
DC
B
colorA ≈ colorB → zA ≈ zB
Each segment is a nodezA ≈ zP, zQ, zS
P Q R
S T
U
A
A
![Page 45: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/45.jpg)
Iteratively solve MRF
![Page 46: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/46.jpg)
Depth through time
![Page 47: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/47.jpg)
MattingInterpolated view without matting
Background Surface
Foreground Surface
Camera
Foreground Alpha
Background
Bayesian MattingChuang et al. 2001
Strip Width
Background
Foreground
![Page 48: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/48.jpg)
Rendering with matting
MattingNo Matting
![Page 49: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/49.jpg)
Representation
Main Layer:
Color
Depth
Main
Boundary
Boundary Layer:
Color
Depth
Alpha
Background
ForegroundStrip Width
![Page 50: Image-based rendering](https://reader036.fdocuments.in/reader036/viewer/2022062722/56813ad2550346895da3017e/html5/thumbnails/50.jpg)
“Massive Arabesque” videoclip