CS395/495: Spring 2003

CS395/495: Spring 2003CS395/495: Spring 2003

IBMR: Image Based IBMR: Image Based Modeling and RenderingModeling and Rendering

Jack Tumblin Jack Tumblin

[email protected]@cs.northwestern.edu

Admin: Admin: How this course worksHow this course works

• Refer to class website: Refer to class website: (soon)(soon)http://www.cs.northwestern.edu/~jet/Teach/2003_3spr_IBMR/IBMRsyllabus2003.htmhttp://www.cs.northwestern.edu/~jet/Teach/2003_3spr_IBMR/IBMRsyllabus2003.htm

• Tasks:Tasks:– Reading, lectures, class participation, projectsReading, lectures, class participation, projects

• Evaluation:Evaluation:– Progressive Programming ProjectProgressive Programming Project– Take-Home Midterm, Take-Home FinalTake-Home Midterm, Take-Home Final

http://www.cs.northwestern.edu/~jet/Teach/2003_3spr_IBMR/IBMRsyllabus2003.htm

GOAL: First-Class PrimitiveGOAL: First-Class Primitive• Want images as ‘first-class’ primitivesWant images as ‘first-class’ primitives

– Useful as BOTH Useful as BOTH inputinput and and outputoutput– ConvertConvert to/from traditional scene descriptions to/from traditional scene descriptions

• Want to mix real & synthetic scenes freelyWant to mix real & synthetic scenes freely• Want to extend photographyWant to extend photography

– Easily capture scene:Easily capture scene:shape, movement, surface/BRDF, lighting …shape, movement, surface/BRDF, lighting …

– Modify & Render the captured scene dataModify & Render the captured scene data• --BUT----BUT-- • images hold only PARTIAL scene informationimages hold only PARTIAL scene information

“ “You can’t always get what you want”You can’t always get what you want” –(Mick Jagger 1968)

Back To Basics: Scene & ImageBack To Basics: Scene & Image

Light + 3D Scene:Light + 3D Scene:Illumination, Illumination,

shape, movement, shape, movement, surface BRDF,… surface BRDF,…

Image PlaneImage PlaneI(x,y)I(x,y)

Ang

le(

Ang

le(

,, ))

Pos

ition

(x,y

)P

ositi

on(x

,y)

2D Image:2D Image:Collection of rays Collection of rays

through a point through a point

Trad. Computer GraphicsTrad. Computer Graphics


Ang

le(

Ang

le(

,, ))

Pos

ition

(x,y

)P

ositi

on(x

,y)





Reduced,Reduced,Incomplete Incomplete InformationInformation

!TOUGH!!TOUGH!‘‘ILL-POSED’ILL-POSED’

Many Simplifications,Many Simplifications,External knowledge…External knowledge…

Trad. Computer VisionTrad. Computer Vision


Ang

le(

Ang

le(

,, ))

Pos

ition

(x,y

)P

ositi

on(x

,y)





2D 2D Display Display Image(s)Image(s)

IBMR Goal: IBMR Goal: BidirectionalBidirectional Rendering Rendering• Both forward and ‘inverse’ rendering!Both forward and ‘inverse’ rendering!

Camera PoseCamera PoseCamera View GeomCamera View GeomScene illuminationScene illuminationObject shape, positionObject shape, positionSurface reflectance,… Surface reflectance,… transparencytransparency

2D 2D Display Display Image(s)Image(s)

2D 2D Display Display Image(s)Image(s)2D 2D

Display Display Image(s)Image(s)IBMRIBMR

TraditionalTraditionalComputerComputerGraphicsGraphics

‘‘3D Scene’ 3D Scene’ DescriptionDescription ‘‘Optical’ Optical’

DescriptionDescription

?! New ?! New Research?Research?

OLDEST IBR: Shadow Maps OLDEST IBR: Shadow Maps (1984)(1984)

Fast Shadows from Z-buffer hardware:Fast Shadows from Z-buffer hardware:1) Make the “Shadow Map”:1) Make the “Shadow Map”:

– Render image seen from light source, Render image seen from light source, BUTBUT– Keep ONLY the Z-buffer values (depth)Keep ONLY the Z-buffer values (depth)

2) Render Scene from Eyepoint:2) Render Scene from Eyepoint:– Pixel + Z depth gives 3D position of surface;Pixel + Z depth gives 3D position of surface;– Project 3D position into Shadow map imageProject 3D position into Shadow map image– If Shadow Map depth < 3D depth, SHADOW!If Shadow Map depth < 3D depth, SHADOW!

Early IBR: QuickTime VR Early IBR: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

1) Four Planar Images 1) Four Planar Images 1 Cylindrical Panorama: 1 Cylindrical Panorama:Re-sampling Required!Re-sampling Required!

Planar Pixels:Planar Pixels: equal distance on x,y plane (tan equal distance on x,y plane (tan-1-1)) Cylinder Pixs: Cylinder Pixs: horiz: equal angle on cylinder (horiz: equal angle on cylinder ())

vert: equal distance on y (tanvert: equal distance on y (tan-1-1))


1) Four Planar Images 1) Four Planar Images 1 Cylindrical Panorama: 1 Cylindrical Panorama:

IN:IN:

OUT:OUT:


2) Windowing, Horizontal-only Reprojection:2) Windowing, Horizontal-only Reprojection:

IN:IN:

OUT:OUT:

View Interpolation: How?View Interpolation: How?• But what if no depth is available?But what if no depth is available?• Traditional Stereo Disparity Map: Traditional Stereo Disparity Map:

pixel-by-pixel search for pixel-by-pixel search for correspondencecorrespondence

View Interpolation: How?View Interpolation: How?• Store Depth at each pixel: reprojectStore Depth at each pixel: reproject• Coarse or Simple 3D model:Coarse or Simple 3D model:

Plenoptic Array: ‘The Matrix Effect’Plenoptic Array: ‘The Matrix Effect’• Brute force!Brute force!

Simple arc, line, or ring array of camerasSimple arc, line, or ring array of cameras• Synchronized shutter Synchronized shutter http://www.http://www.ruffyruffy.com/.com/firinglinefiringline.html.html

• Warp/blend between images to Warp/blend between images to change viewpoint on ‘time-frozen’ scene:change viewpoint on ‘time-frozen’ scene:

http://www.ruffy.com/firingline.html





• for a given scene, describe for a given scene, describe ALL rays through ALL rays through – ALLALL pixels, of pixels, of – ALLALL cameras, at cameras, at – ALLALL wavelengths, wavelengths, – ALLALL time time

F(F(x,y,zx,y,z,, ,,,, , , tt))““Eyeballs Everywhere” Eyeballs Everywhere”

function (5-D x 2-D!)function (5-D x 2-D!)

Plenoptic Function Plenoptic Function (Adelson, Bergen `91)(Adelson, Bergen `91)

………………………………

…………………………

Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96

• http://www.cs. http://www.cs.washingtonwashington.edu/homes/.edu/homes/seitzseitz//vmorphvmorph//vmorphvmorph..htmhtm1)Manually set some 1)Manually set some corresp.points corresp.points (eye corners, etc.)(eye corners, etc.)

2) pre-warp and2) pre-warp andpost-warp to matchpost-warp to matchpoints in 3D,points in 3D,

3) Reproject for3) Reproject forVirtual camerasVirtual cameras

http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm










Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96

• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

http://www.cs.cmu.edu/~seitz/vmorph/mfig1.mpg

‘‘Scene’ causes Light FieldScene’ causes Light FieldLight field: holds all Light field: holds all outgoingoutgoing light rays light rays

Shape, Shape, Position,Position,Movement,Movement,

BRDF,BRDF,Texture,Texture,ScatteringScattering

Emitted Emitted LightLight

Reflected,Reflected,Scattered,Scattered,Light …Light … Cameras capture Cameras capture

subset of these subset of these rays.rays.

? Can we recover ? Can we recover Shape Shape ??Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?

Ray colors might Ray colors might not match for not match for non-diffuse non-diffuse materials (BRDF)materials (BRDF)

? Can we recover ? Can we recover Surface Material Surface Material ??Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?

Ray colors might Ray colors might not match for not match for non-diffuse non-diffuse materials (BRDF)materials (BRDF)

Hey, wait…Hey, wait…• Light field describes light Light field describes light LEAVINGLEAVING the the

enclosing surface….enclosing surface….• ? Isn’t there a complementary ‘light field’ for ? Isn’t there a complementary ‘light field’ for

the light the light ENTERINGENTERING the surface? the surface?

• *YES* ‘Image-Based Lighting’ too!*YES* ‘Image-Based Lighting’ too!

• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere

around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere

around object/scenearound object/scene

F(F(xxcc,y,ycc,,cc,,cc,x,xll,y,yll ll,,ll,, , t, t))

‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))

cameracamera




F(F(xxcc,y,ycc,,cc,,cc,x,xll,y,yll ll,,ll,, , t, t))


cameracamera

cc

cc




F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))


cameracamera

Projector (laser brick)Projector (laser brick)



around object/scenearound object/scene– (and wavelength and time)(and wavelength and time)

F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))


cameracamera

projectorprojector


• ! Complete !! Complete !– Geometry, Lighting, BRDF,… Geometry, Lighting, BRDF,… ! NOT REQUIRED !! NOT REQUIRED !

• Preposterously Huge: Preposterously Huge: (?!?! (?!?! 8-D8-D function sampled at image resolution !?!?), function sampled at image resolution !?!?), but but

• Hugely Redundant:Hugely Redundant:– WavelengthWavelengthRGB triplets, ignore Time, and Restrict RGB triplets, ignore Time, and Restrict

eyepoint movement: maybe ~3 or 4D ?eyepoint movement: maybe ~3 or 4D ?– Very Similar images—use Warping rules? (SIGG2002)Very Similar images—use Warping rules? (SIGG2002)– Exploit Movie Storage/Compression Methods?Exploit Movie Storage/Compression Methods?

IBR-Motivating OpinionsIBR-Motivating Opinions““Computer Graphics: Hard”Computer Graphics: Hard”

– Complex!Complex! geometry, texture, lighting, shadows, geometry, texture, lighting, shadows, compositing, BRDF, interreflections, etc. etc., etc., …compositing, BRDF, interreflections, etc. etc., etc., …

– Irregular!Irregular! Visibility,Topology, Render Eqn.,… Visibility,Topology, Render Eqn.,…– Isolated!Isolated! Tough to use real objects in CGI Tough to use real objects in CGI– Slow!Slow! compute-bound, off-line only,… compute-bound, off-line only,…

““Digital Imaging: Easy”Digital Imaging: Easy”– Simple!Simple! More quality? Just pump more pixels! More quality? Just pump more pixels!– Regular!Regular! Vectorized, compressible, pipelined… Vectorized, compressible, pipelined…– Accessible!Accessible! Use real OR synthetic (CGI) images! Use real OR synthetic (CGI) images!– Fast!Fast! Scalable, Image reuse, a path to interactivity… Scalable, Image reuse, a path to interactivity…

Practical IBMRPractical IBMR

What useful partial solutions are possible?What useful partial solutions are possible?

• Texture Maps++: Panoramas, Env. MapsTexture Maps++: Panoramas, Env. Maps• Image(s)+Depth: (3D shell) Image(s)+Depth: (3D shell) • EstimatingEstimating Depth & Recovering Silhouettes Depth & Recovering Silhouettes• ‘‘Light Probe’ measures real-world lightLight Probe’ measures real-world light• Structured Light to Recover Surfaces…Structured Light to Recover Surfaces…• Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

ConclusionConclusion

• Heavy overlap with computer vision: Heavy overlap with computer vision: careful not to re-invent & re-name!careful not to re-invent & re-name!

• Elegant Geometry is Elegant Geometry is at the heart of it allat the heart of it all, , even surface reflectance, illumination, etc. even surface reflectance, illumination, etc. etc.etc.

• THUS: we’ll dive into geometry--all the rest THUS: we’ll dive into geometry--all the rest is built on it! is built on it!

CS395/495: Spring 2003

Documents

Transcript of CS395/495: Spring 2003