Primal-Dual Codingto Probe Light Transport

Matthew O’TooleUniversity of Toronto

Ramesh RaskarMIT Media Lab

Kyros KutulakosUniversity of Toronto

conventional photography

photo records all light paths that reach camera sensor

primal-dual coding photography

photo only records a user-defined subset of light paths

conventional photography

primal-dual coding photography

new photography regime: primal-dual codingmodulate contribution of specific light path

contributions

transport probing equationgeneralizes conventional photography & confocal imaging [Wilson et al. 96]

combine illumination coding [Schechner et al. 07] & exposure coding [Hitomi et al. 11]

efficient primal-dual codes by stochastic matrix probingprovides convergence & optimality guarantees [Bekas et al. 07]

1. the result of light transport from sources to sensor pixels

2. the image recorded by the camera’s sensor

what is a photo?

transport matrix [Ng et al. 03]photo

light

conventional photography

degrees of freedom

primal-dual coding photography

“probing matrix”

degrees of freedom

multiply

primal-dual coding photography

“probing matrix”

degrees of freedom

primal-dual coding photography

“probing matrix”

degrees of freedom

primal-dual coding photography

degrees of freedom

mathematical formulation of primal-dual coding photography

one-shot primal-dual coding photography

experimental setup

SLR camera LCD mask

projector

scene

beamsplitter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

step 2illuminate scene

with vector . (primal code)

step 3attenuate image with vector .

(dual code)

step 4repeat times

step 1open shutter

step 5close shutter

Rademacher primal-dual codes

stochastic diagonal estimator [Bekas et al. 07]

primal codes are Rademacher random vectors: random vector in

dual codes derive from primal code:

codes converge to identity probing matrix:

variance of pixel for primal-dual codes

aperture correlation (microscopy) is a diagonal estimator [Wilson et al. 96, Levoy et al. 04]

stochastic estimator for general probingdual codes for general probing matrix :

designing probing matrices

designing probing matrices

transport matrix

designing probing matrices

transport matrix

designing probing matrices

camera

pro

jector

transport matrix

designing probing matrices

camera

pro

jector

camera

pro

jector

transport matrix

designing probing matrices

camera

pro

jector

camera

pro

jector

transport matrix

designing probing matrices

camera

pro

jector

camera

pro

jector

transport matrix

coaxial example: contrast-enhancing direct light

conventionalphoto

all light paths

coaxial example: contrast-enhancing direct light

direct +½ indirect

conventionalphoto

direct + 1/2 indirect light paths

coaxial example: contrast-enhancing direct light

direct +½ indirect

conventionalphoto

direct +¼ indirect

direct +1/16 indirect

direct + 1/16 indirect light paths

coaxial example: capturing short to long range paths

conventional

all light paths

coaxial example: capturing short to long range paths

conventional

indirect

indirect light paths

coaxial example: capturing short to long range paths

conventional

indirect mid-range indirect

medium to long range indirect light paths

coaxial example: capturing short to long range paths

conventional

indirect mid-range indirect

long-range indirect

long range indirect light paths

coaxial example: separating light transport effects

conventional

all light paths

coaxial example: separating light transport effects

conventional

indirect

indirect light paths

coaxial example: separating light transport effects

conventional

indirect direct + backscatter

direct + back-scatter light paths

coaxial example: separating light transport effects

indirect [Nayar et al. 06]

indirect direct + backscatter

direct [Nayar et al. 06]

coaxial example: separating light transport effects

indirect direct + backscatter

low-freq. indirect high-freq. indirect

conventional

all light paths

coaxial example: de-scattering [Fuchs et al. 08]

conventional de-scattered image

de-scattered diagonal

coaxial example: de-scattering [Fuchs et al. 08]

coaxial example: de-scattering [Fuchs et al. 08]

conventional

direct + 1/16 indirect

direct + 1/16 indirect light paths

de-scattered image

conventional

direct + 1/16 indirect shifted indirect

de-scattered image

coaxial example: de-scattering [Fuchs et al. 08]

shifted indirect + 1/16 light paths

non-coaxial example: making 3D regions invisible

all light paths

“privacy zone”

non-coaxial example: making 3D regions invisible

non-coaxial example: making 3D regions invisible

non-coaxial example: making 3D regions invisible

non-coaxial example: making 3D regions invisible

privacy zone

non-coaxial example: making 3D regions invisible

non-coaxial example: color coding regions of space

color coded zones

non-coaxial example: color coding regions of space

non-coaxial example: color coding regions of space

real-time PDC photographymatching 1440Hz DLP kits for illumination and attenuation

real-time PDC photography

projector

matching 1440Hz DLP kits for illumination and attenuation

real-time PDC photography

mask

camera

matching 1440Hz DLP kits for illumination and attenuation

DMD

additional details on primal-dual coding

non-negative primal-dual codesmany cases realized with strictly non-negative codesfor all others, subtract two photos computationally

codes robust to optical misalignmentsaccount for both spatial and temporal misalignments

one-shot versus multi-shotone-shot limited by projector display ratemulti-shot limited by camera read noise

concluding remarks

primal-dual coding photographyproduces new types of photos governed by probing equation

combines coded illumination and coded exposure

guarantees optimality and convergence rate of codes

can be a see-through device

future worknew possibilities for optical applications

combine with existing computational photography techniques (image-based relighting,geometry acquisition, …)

Matthew O’TooleUniversity of Toronto

Ramesh RaskarMIT Media Lab

Kyros KutulakosUniversity of Toronto

Primal-Dual Codingto Probe Light Transporthttp://www.dgp.toronto.edu/~motoole/primaldualcoding.html

64 codes

16 codes 32 codes8 codes

128 codes 256 codes