PowerPoint Presentation

Light FieldsProperties and applicationsOutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport

OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport

The Plenoptic Function

Plenus Complete, full. Optic - appearance, look.The set of things one can ever see

Light intensity as a function ofViewpoint orientation and positionTimeWavelength7D function!The 5D Plenoptic FunctionIgnoring wavelength and time We need a 5D function to describe light rays across occlusions2D orientation3D position

5The Light Field (4DAssuming no occlusions Light is constant across raysNeed only 4D to represent the space of RaysIs this assumption reasonable?In free space, i.e outside the convex hull of the scene occluders

Light field is the basic mathematical object in computational photography, graphics etc.6The Light FieldParameterizationsPoint on a Plane or curved Surface (2D) and Direction on a Hemisphere (2D)Two Points on a SphereTwo Points on two different Planes

Two Plane ParameterizationConvenient parameterization for computational photographyWhy?Similar to camera geometry (i.e. film plane vs lens plane)Linear parameterization - easy computations , no trigonometric functions, etc.

2D light fieldUsed for visualization. Assume the world is flat (2D)

The image a pinhole at (u,v) capturesAll views of a pixel (s,t)IntuitionLight Field Rendering , Levoy Hanrahan '96.10OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport

Acquisition of Light FieldsSynthetic 3D Scene

Discretize s,t,u,v and capture all rays intersecting the objects using a standard Ray Tracer

12Acquisition of Light Fields

Real world scenesWill be explained in more detail next week13OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport

Changing the View PointProblem: Computer GraphicsRender a novel view point without expensive ray tracingSolution:Sample a Synthetic light field using Ray TracingUse the Light Field to generate any point of view, no need to Ray Trace

Light Field Rendering , Levoy Hanrahan '96. view , views15Changing the View PointConceptually: Use Ray Trace from all pixels in image plane

Actually: Use Homographic mapping from XY plane to the VU and TS, and lookup resulting ray radiance.

pinholeLight Field Interpolation

NNNN + LinearLinearChanging the focal plane

Fourier Slice Photography , Ng, 05The goal, given a light field, generate views with different focal planes.18In-Camera Light Field Parameterization

We have an in camera light field parameterized by the lens plane and the sensor plane. Moving the sensor, i.e the focal plane, is equivalent to changing the image plane. The depth at which points appear focused.19The camera operatorCan define a camera as an operator on the Light Field.The conventional camera operator:

yx [Stroebel et al. 1986]A camera may be represented as an operator on the light field. For example the conventional camera. Phi is angle between the sensor plane normal and the incoming ray (x,y,u,v)20Reminder - Thin lens formulaDD1DD1const+=To focus closer - increase the sensor-to-lens distance.21Refocusing - Reparameterization

In order to get a light field with a new focal plane we reparametrize the light field which is equivalent to a shear of the light field.22Reparametrization - 4D

Refocusing - ReparameterizationRefocusReparameterization of the light fieldShearing of the Light field

Refocusing camera operatorShear and Integrate the original light field

*(cos term from conventional camera model is absorbed into L)

25Computation of Refocusing OperatorNave ApproachFor every X,Y go over all U,V and calculate the sum after reparameterization => O(n^4)

Can we do better ????

yx

26Fourier Analysis of the Camera OperatorProjection in the spatial domain Slicing in the fourier domainGiven that:

Then:

27Fourier Slice TheoremF Fourier Transform OperatorI Integral Projection OperatorS Slicing Operator

Known from CT in 2DApplicable (non trivially to N,M)Specifically for 4D , our light field28Fourier Analysis of the Camera Operator

Immediate result Faster Algorithm!29

Fourier Slice Photography , Ng, 05Fourier Slice Photography Thm More corollaries Two important results that are worth mentioning:1. Filtered Light Field Photography Thm

2. The light field dimensionality gap

Filtered Light Field Photography ThmTheorem: Filtered Light Field Photography

Note that when we do the refocusing were not really working on the actual light field, but on a filtered (sampled) representation of it.32The light field dimensionality gap

The light field is 4DIn the frequency domain The support of all the images with different focus depth is a 3D manifold

This observation was used in order to generate new views of the scene from a focal stack (Levin et al. 2010)

OutlineWhat are light fields Acquisition of light fields from a 3D scenefrom a real world sceneImage rendering from light fieldsChanging viewing angleChanging the focal plane Sampling and reconstructionDepth vs spectral supportOptimal reconstructionAnalysis of light transport

Sampling and reconstruction of light fieldsIn many cases the sampling rate is bounded due to camera limitations.We want to understand the spectrum of the light fields better in order to reconstruct betterTwo main papers go in this directionPlenoptic sampling (SIGGRAPH 2000) light field spectrum VS. scenes depthFrequency analysis of light transport (SIGGRAPH 2005) - light field spectrum VS. Path of the lightWe need to understand the spectrum of the light field in order to understand how to sample/reconstruct the acquired light field

35Light Field SamplingLight Field Acquisition DiscretizationLight Field Sampling is LimitedExample Camera Array:

u,vt,s36Sampling in frequency domainAliasing in the frequency domain

Need to analyze Light Field Spectrum

*=

ALIASING!

No Aliasing! Low pass filter . ( )37Scene Depth and Light FieldLight Field Spectrum is related to Scene DepthFrom Lambertian property each point in the scene corresponds to a line in the Light FieldLine slope is a function of the depth (z) of the point.

Plenoptic Sampling , Chai et al., 00.38Spectral Support of Light FieldConstant DepthSceneLight Field

LF SpectrumPlenoptic Sampling , Chai et al., 00.39Spectral Support of Light FieldVarying Depth

SceneLF SpectrumPlenoptic Sampling , Chai et al., 00.Spectral Support of Light Field

Plenoptic Sampling , Chai et al., 00.41Reconstruction Filters

Optimal Slope for filter:

Plenoptic Sampling , Chai et al., 00.LimitationsAssumptionsLambertian surfaces

Free Space No occlusions

Frequency Analysis of Light TransportInformally: Different features of lighting and scene causes different effects in the Frequency Content Blurry Reflections

Shadow Boundries

Low frequencyHigh frequencyA Frequency analysis of Light Transport , Durand et al. 05.Not Wave Optics!!!

Frequency Analysis of Light TransportLook at light transport as a signal processing system.Light source is the input signalInteraction are filters / transforms

SourceTransportOcclusionTransportReflection(BRDF)Local Light FieldWe study the local 4D Light Field around a central Ray during transportIn Spatial DomainIn Frequency Domain

* Local light field offers us the ability to talk about the Spectrum In a local setting

47Local Light Field (2D) Parameterization The analysis is in flatland, an extension to 4D light field is available

x-v parameterizationx- parameterizationA Frequency analysis of Light Transport , Durand et al. 05.Example Scenario

ReflectionA Frequency analysis of Light Transport , Durand et al. 05.Light Transport Spatial DomainLight Propagation Shear of the local Light FieldNo change in slope (v)Linear change in displacement (X)

Light Transport Frequency DomainShear in spatial domain is also a shear in Frequency domain

OcclusionSpatial domain:Occlusion pointwise multiplication in the spatial domainThe incoming light field is multiplied by the binary occlusion function of the occluders.

Frequency domain convolution in the frequency domain:

Occlusion example

53Reflection

* Similar analysis for curved surfaces is also presented in the paper54Reflection - cosine termSpatial domain - multiplication::

Frequency domain:

cosine term exampleIncoming Light fieldLight fieldAfter cosine term

56Reflection Mirror reparameterization

Reparameterization exampleIncoming Light fieldLight fieldAfter reparameterization

Reflection - BRDFBRDF Intuition*=directiondirectionx (space)x (space)60Reflection - BRDFBRDF example

Incoming Light fieldLight fieldAfter BRDF change

-