Procam and Campro

Shree K. Nayar

Computer ScienceColumbia University

Support: NSF, ONR

Procams 2006PROCAMS Shree K. Nayar,

Convergence of Cameras and Projectors

Cameras for Projectors (Campro):

Display better or new images

(Geometry, Shadows, Radiometry, Focus, Multi-Projectors)

P

C

Projectors & Cameras for Apps (ProCamApp):

Enhance existing or create new application

(Immersion, Augmentation, Navigation, Surgery)

P

C

Projectors for Cameras (Procam):

Capture better or new images

(Quality, Matting, Relighting, Refocusing)

P

C

PROCAMS Shree K. Nayar,

Brief History of the Flash

Gas FlameIbbetson

1840

FlashbulbChauffour

1893

ElectricFlashbubVierkotter

1925

CommercialFlashbub

Ostermeier

1930

DigitalCameraFlash

2006

Flash/No Flash

Multi-Flash

Flash Matting

PROCAMS Shree K. Nayar,

Projector as a Programmable Flash

Lighting Capture

2D 2D

4D 4D

PROCAMS Shree K. Nayar,

Projector as a Programmable Flash

Depth from Projector Defocus

Image-Complete Depth Maps

(with Li Zhang, Siggraph 2006)

Separation of Direct and Global Illumination

Scattering Properties of Real Materials

(with Krishnan, Grossberg, Raskar, Siggraph 2006)

PROCAMS Shree K. Nayar,

Courtesy of Koesbong @ flickr.com

Camera Defocus

PROCAMS Shree K. Nayar,

Focal Plane SurfaceCamera

Depth from Camera Defocus

( Pentland 87, Subbarao 88, Darrell and Wohn 88,Nayar and Nakagawa 89, Ens and Lawrence 91, Bove 93, Nayar et al. 95, Watanabe et al. 98 )

Camera Defocus:

Captured

PROCAMS Shree K. Nayar,

Focal Plane

Lamp

Projector

Projector Defocus

PROCAMS Shree K. Nayar,

Focal Plane Surface Projector

Lamp

Projector Defocus

PROCAMS Shree K. Nayar,

Focal Plane Surface Projector

Lamp

Projector Defocus

Projector Defocus:

Projected

PROCAMS Shree K. Nayar,

Camera Defocus:

Captured

Camera Defocus vs. Projector Defocus

Projector Defocus:

Projected

PROCAMS Shree K. Nayar,

Focal Plane

LampSh

ift

Surface ProjectorCamera

Projector Defocus Model:

Temporal Defocus Method

Spectrum

Depth Estimation:PROCAMS Shree K. Nayar,

Projector

Half-mirror

Camera

Half-mirror

Projector

Camera

Experimental Setup

PROCAMS Shree K. Nayar,

PROCAMS Shree K. Nayar,

Scene Depth Map

Fence and Leaves: Complex Occlusions

PROCAMS Shree K. Nayar,

Scene Depth Map

3D Model

Fence and Leaves: Complex Occlusions

PROCAMS Shree K. Nayar,

Refocusing

PROCAMS Shree K. Nayar,

Scene Depth Map

Playing Cards: Object Insertion

PROCAMS Shree K. Nayar,

Scene Depth Map

Green Screen Capture Depth-based Composition

Playing Cards: Object Insertion

PROCAMS Shree K. Nayar,

Projector’s Narrow Depth of Field

PROCAMS Shree K. Nayar,

Multiple Projectors: (Bimber and Emmerling 2006)

Focused Projection at Multiple Depths

PROCAMS Shree K. Nayar,

Single Projector:

Focused Projection at Multiple Depths

PROCAMS Shree K. Nayar,

Projector Defocus: Three Plane Example

15 inches15 inches

PROCAMS Shree K. Nayar,

CompensatedProjection

Front Plane Middle Plane Back Plane

Original Image

Normal Projection

Projector Defocus Compensation: Results

PROCAMS Shree K. Nayar,

Projector as a Programmable Flash

Depth from Projector Defocus

Image-Complete Depth Maps

(with Li Zhang, Siggraph 2006)

Separation of Direct and Global Illumination

Scattering Properties of Real Materials

(with Krishnan, Grossberg, Raskar, Siggraph 2006)

source

surface

P

Direct and Global Illumination

A

A : Direct

B

B : Interrelection

C

C : Subsurface

D

participating medium

D : Volumetric translucent surface

E

E : Diffusion

camera

Related Work

(Seitz et. al., ICCV 05)

• Inverse Light Transport

(Sen et. al., Siggraph 05)

T

• Dual Photography

Fast Separation of Direct and Global Images

PROCAMS Shree K. Nayar,

],[],[],[ icLicLicL gd

direct globalradiance

Direct and Global Components: Interreflections

surface

i

camera

source

P

g jiLjiAicL ],[],[],[

j

BRDF and geometryPROCAMS Shree K. Nayar,

High Frequency Illumination Pattern

surface

camera

source

fraction of activated source elements

],[],[],[ icLicLicL gd +

i

PROCAMS Shree K. Nayar,

High Frequency Illumination Pattern

surface

fraction of activated source elements

camera

source

],[],[],[ icLicLicL gd + - ],[],[ icLicL g )1(

i

PROCAMS Shree K. Nayar,PROCAMS Shree K. Nayar,

:2

1 min2LLg

Separation from Two Images

direct global

,minmax LLLd

PROCAMS Shree K. Nayar,

Other Global Effects

• Subsurface Scattering

• Volumetric Scattering

• Diffusion

PROCAMS Shree K. Nayar,

Diffuse Interreflections

SpecularInterreflections

Volumetric Scattering Subsurface

Scattering

Diffusion

PROCAMS Shree K. Nayar,

Scene

Direct GlobalPROCAMS Shree K. Nayar,

Real World Examples:

Can You Guess the Images?

PROCAMS Shree K. Nayar,

Eggs: Diffuse Interreflections

Direct GlobalPROCAMS Shree K. Nayar,

Wooden Blocks: Specular Interreflections

Direct GlobalPROCAMS Shree K. Nayar,

Novel Images

PROCAMS Shree K. Nayar,

Photometric Stereo using Direct Images

Bowl

Shape

Source 1 Source 2 Source 3

Nayar et al., 1991

Direct

Global

PROCAMS Shree K. Nayar,

Peppers: Subsurface Scattering

Direct GlobalPROCAMS Shree K. Nayar,

Novel Images

PROCAMS Shree K. Nayar,

Hand

Direct Global

Skin: Hanrahan and Krueger

93,

Uchida 96, Haro 01, Jensen et

al. 01,

Cula and Dana 02, Igarashi et

al. 05, Weyrich et al. 05

PROCAMS Shree K. Nayar,

Separation from a Single Image

PROCAMS Shree K. Nayar,

Face

Direct Global

SumPROCAMS Shree K. Nayar,

Skin Tone Control

Skin Color and Lipids:

Tsumura et al. 03

PROCAMS Shree K. Nayar,

Summary

Coded Structured Light Shifted Sinusoids

Depth from Projector Defocus

(Minimum Images: 3)

Direct-Global Separation

(Minimum Images: 2)

PROCAMS Shree K. Nayar,

Open Questions

• What is Recoverable with a Projector and a Camera ?

• How Many Illumination Patterns?

• What are the Illumination Patterns?

PROCAMS Shree K. Nayar,

Cameras for Projectors Projectors for Cameras

Convergence of Cameras and Projectors

Projectors in 2006

SizePowerCost

x 1/10x 1/10x 1/10

PROCAMS Shree K. Nayar,