Computational Photography Light Field Rendering Jinxiang Chai.

51
Computational Photography Light Field Rendering Jinxiang Chai
  • date post

    21-Dec-2015
  • Category

    Documents

  • view

    219
  • download

    1

Transcript of Computational Photography Light Field Rendering Jinxiang Chai.

Page 1: Computational Photography Light Field Rendering Jinxiang Chai.

Computational PhotographyLight Field Rendering

Jinxiang Chai

Page 2: Computational Photography Light Field Rendering Jinxiang Chai.

Image-based Modeling: Challenging Scenes

Why will they produce poor results?

- lack of discernible features

- occlusions

- difficult to capture high-level structure

- illumination changes

- specular surfaces

Page 3: Computational Photography Light Field Rendering Jinxiang Chai.

Some Solutions

- Use priors to constrain the modeling space

- Aid modeling process with minimal user interaction

- Combine image-based modeling with other modeling approaches

Page 4: Computational Photography Light Field Rendering Jinxiang Chai.

Videos

Morphable face (click here)

Image-based tree modeling (click here)

Video trace (click here)

3D modeling by ortho-images (Click here)

Page 5: Computational Photography Light Field Rendering Jinxiang Chai.

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Light field

Images + Depth

Geometry+ Materials

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Page 6: Computational Photography Light Field Rendering Jinxiang Chai.

Outline

Light field rendering [Levoy and Hanranhan SIG96]

3D light field (concentric mosaics) [Shum and He Sig99]

Page 7: Computational Photography Light Field Rendering Jinxiang Chai.

Plenoptic Function

Can reconstruct every possible view, at every moment, from every position, at every wavelength

Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality!

An image is a 2D sample of plenoptic function!

P(x,y,z,θ,φ,λ,t)

Page 8: Computational Photography Light Field Rendering Jinxiang Chai.

Ray

Let’s not worry about time and color:

5D• 3D position

• 2D direction

P(x,y,z,)

Page 9: Computational Photography Light Field Rendering Jinxiang Chai.

Static object Camera

No Change in

Radiance

Static Lighting

How can we use this?

Page 10: Computational Photography Light Field Rendering Jinxiang Chai.

Static object Camera

No Change in

Radiance

Static Lighting

How can we use this?

Page 11: Computational Photography Light Field Rendering Jinxiang Chai.

Ray Reuse

Infinite line• Assume light is constant (vacuum)

4D• 2D direction

• 2D position

• non-dispersive medium

Slide by Rick Szeliski and Michael Cohen

Page 12: Computational Photography Light Field Rendering Jinxiang Chai.

Only need plenoptic surface

Page 13: Computational Photography Light Field Rendering Jinxiang Chai.

Synthesizing novel views

Assume we capture every ray in 3D space!

Page 14: Computational Photography Light Field Rendering Jinxiang Chai.

Synthesizing novel views

Page 15: Computational Photography Light Field Rendering Jinxiang Chai.

Light field / Lumigraph

Outside convex space

4DStuff

Empty

Page 16: Computational Photography Light Field Rendering Jinxiang Chai.

Light Field

How to represent rays?

How to capture rays?

How to use captured rays for rendering

Page 17: Computational Photography Light Field Rendering Jinxiang Chai.

Light Field

How to represent rays?

How to capture rays?

How to use captured rays for rendering

Page 18: Computational Photography Light Field Rendering Jinxiang Chai.

Light field - Organization

2D position

2D direction

s

Page 19: Computational Photography Light Field Rendering Jinxiang Chai.

Light field - Organization

2D position

2D position

2 plane parameterization

su

Page 20: Computational Photography Light Field Rendering Jinxiang Chai.

Light field - Organization

2D position

2D position

2 plane parameterization

us

t s,tu,v

v

s,t

u,v

Page 21: Computational Photography Light Field Rendering Jinxiang Chai.

Light field - Organization

Hold u,v constant

Let s,t vary

What do we get?

s,tu,v

Page 22: Computational Photography Light Field Rendering Jinxiang Chai.

Lumigraph - Organization

Hold s,t constant

Let u,v vary

An image

s,tu,v

Page 23: Computational Photography Light Field Rendering Jinxiang Chai.

Lightfield / Lumigraph

Page 24: Computational Photography Light Field Rendering Jinxiang Chai.

Light field/lumigraph - Capture

Idea 1• Move camera carefully over u,v

plane

• Gantry> see Light field paper

s,tu,v

Page 25: Computational Photography Light Field Rendering Jinxiang Chai.

Stanford multi-camera array

640 × 480 pixels ×30 fps × 128 cameras

synchronized timing

continuous streaming

flexible arrangement

Page 26: Computational Photography Light Field Rendering Jinxiang Chai.

For each output pixel• determine s,t,u,v

• either• use closest discrete RGB• interpolate near values

s u

Light field/lumigraph - rendering

Page 27: Computational Photography Light Field Rendering Jinxiang Chai.

Light field/lumigraph - rendering

Nearest• closest s

• closest u

• draw it

Blend 16 nearest• quadrilinear interpolation

s u

Page 28: Computational Photography Light Field Rendering Jinxiang Chai.

Ray interpolation

s u

Nearest neighbor

Linear interpolation in S-T

Quadrilinear interpolation

Page 29: Computational Photography Light Field Rendering Jinxiang Chai.

Image Plane

Camera Plane

Light FieldLight Field CaptureCapture RenderingRendering

Light Field/Lumigraph Rendering

Page 30: Computational Photography Light Field Rendering Jinxiang Chai.

Light fields

Advantages:• No geometry needed

• Simpler computation vs. traditional CG

• Cost independent of scene complexity

• Cost independent of material properties and other optical effects

Disadvantages:• Static geometry

• Fixed lighting

• High storage cost

Page 31: Computational Photography Light Field Rendering Jinxiang Chai.

3D plenoptic function

Image is 2D

Light field/lumigraph is 4D

What happens to 3D?

- 3D light field subset

- Concentric mosaic [Shum and He]

Page 32: Computational Photography Light Field Rendering Jinxiang Chai.

3D light field

One row of s,t plane• i.e., hold t constant

s,t u,v

Page 33: Computational Photography Light Field Rendering Jinxiang Chai.

3D light field

One row of s,t plane• i.e., hold t constant

• thus s,u,v

• a “row of images”

s

u,v

Page 34: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics [Shum and He]

Polar coordinate system:

- hold r constant

- thus (θ,u,v)

Page 35: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

Why concentric mosaic?

- easy to capture

- relatively small in storage size

Page 36: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

How to captured images?

Page 37: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

How to render a new image?

Page 38: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

How to render a new image?

- for each ray, retrieval the closest captured rays

Page 39: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

How to render a new image?

- for each ray, retrieval the closest captured rays

Page 40: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

How to render a new image?

- for each ray, retrieval the closest captured rays

Page 41: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object

How to retrieval the closest rays?

Page 42: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Page 43: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Page 44: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Page 45: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Page 46: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object (s,t) interpolation plane

How to synthesize the color of rays?

Page 47: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above object (s,t) interpolation plane

How to synthesize the color of rays? - bilinear interpolation

Page 48: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

Page 49: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

From above

Page 50: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

What are limitations?

Page 51: Computational Photography Light Field Rendering Jinxiang Chai.

Concentric mosaics

What are limitations? - limited rendering region?

- large vertical distortion