Raskar Banff

Mitsubishi Electric Research Laboratories Raskar 2007

Media Lab, MIT

Cambridge, MA

From 4D Capture to 6D Display: From 4D Capture to 6D Display: A mask-based approachA mask-based approach

Ramesh Raskar

P ro jec tor

P os =0

P os =255

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006

Discussion Topics

• What is the info content of a 3D scene?– Encoding appearance and geometric

complexity

• What are the dimensions beyond viewpt?– Lighting?

• What other optical sensors we can use?

R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J

Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns

Siggraph 2007

Prakash: Lighting-Aware Motion Capture UsingPhotosensing Markers and Multiplexed Illuminators

Vicon Motion Capture

High-speed IR Camera

Medical Rehabilitation Athlete Analysis

Performance Capture Biomechanical Analysis

Imperceptible Tags under clothing, tracked under ambient light

Hidden Marker Tags

Outdoors

Unique Id

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006Labeling Space(Indoor GPS)

Each location receives a

unique temporal code

But 60Hz video projector

is too slow

Projector

Pos=255

Pattern

For each taga. From light sequence, decode x and y

coordinateb. Transmit back to RF reader (Id, x, y)

0 1 1 0 0 X=12

Inside of Multi-LED Emitter

Analog Space Labeling

Multi-LED

Beacon1

Beacon2

Beacon3

Imperceptible Tags Location

Location Orientation

3D Overlay Orientation

Imperceptible Tags Incident Illumination

Inverse Optical Mo-Cap

High Speed Camera Detect blobs in each frame

Reflective/Emitting Marker Disambiguate in camera Only Location

High Speed Projector Label the 3D space

Photosensing Marker Find ego-position Location, Orientation, Illum

On-set MoCap: Location + Orientation + Incident Illumination

Coded Illumination Sensor Skin• 500 Hz with Id for each Marker Tag• Capture in Natural Environment

– Visually imperceptible tags– Photosensing Tag can be hidden under clothes– Ambient lighting is ok

• Unlimited Number of Tags– Light sensitive fabric for dense sampling

• Non-imaging, complete privacy• Base station and tags only a few 10’s $

• Body scan + bio– Elderly, patients, athletes, performers

Project Topics

• Structured Light Scanning– Fast Stripping

• Can you build a scanner using very low cost hardware?• Without full 2D cameras or video projectors?

– Global-direct Separation• Can you scan difficult (global effect) using direct/global

separation?

Towards a 6D Display

Passive Reflectance Field Display

Martin Fuchs, Ramesh Raskar,Hans-Peter Seidel, Hendrik P. A. Lensch

Siggraph 2008

1 MPI Informatik, Germany 2 MIT

Martin Fuchs <mfuchs@mpi-inf.mpg.de>

Improved Design

Variance with Observer

recall:

Observer-Variance

6D Construction

Illumination + Spatial Variation

Variance with Observation Angle

Towards 6D

6D Results

Future Work

• Efficient manufacturing• scale• precision• How fine can we get our structures?

– is 6D really practical?• Extensions for local illumination ?

Coded Aperture CameraCoded Aperture Camera

The aperture of a 100 mm lens is modified

Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern

In Focus Photo

Out of Focus Photo: Open Aperture

Out of Focus Photo: Coded Aperture

Captured Blurred Photo

Refocused on Person

Mask? SensorMask

SensorMask? Sensor

MaskSensor

Mask? Sensor

4D Light Field from 2D Photo:

Heterodyne Light Field Camera

Full Resolution Digital Refocusing:

Coded Aperture Camera

Light Field Inside a CameraLight Field Inside a Camera

Lenslet-based Light Field cameraLenslet-based Light Field camera

[Adelson and Wang, 1992, Ng et al. 2005 ]

Light Field Inside a CameraLight Field Inside a Camera

Stanford Plenoptic Camera Stanford Plenoptic Camera [Ng et al 2005][Ng et al 2005]

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125μ square-sided microlenses

Digital RefocusingDigital Refocusing

[Ng et al 2005][Ng et al 2005]

Can we achieve this with a Can we achieve this with a MaskMask alone? alone?

Mask based Light Field CameraMask Sensor

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

How to Capture 4D Light Field with 2D

Sensor ?

What should be the pattern of the mask ?

Radio Frequency HeterodyningRadio Frequency Heterodyning

Baseband Audio Signal

Receiver: DemodulationHigh Freq Carrier 100 MHz

ReferenceCarrier

Incoming Signal

99 MHz

Optical HeterodyningOptical Heterodyning

Photographic Signal

(Light Field)

Carrier Incident Modulated

SignalReference

Carrier

Main LensObject Mask Sensor

RecoveredLight Field

Software Demodulation

Baseband Audio Signal

Receiver: DemodulationHigh Freq Carrier 100 MHz

ReferenceCarrier

Incoming Signal

99 MHz

Captured 2D Photo

Encoding due to Mask

2D FFT

Traditional Camera Photo

Heterodyne Camera Photo

Magnitude of 2D FFT

2D FFT

Magnitude of 2D FFT

Computing 4D Light Field2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800

2D FFT

Rearrange 2D tiles into 4D planes200*200*9*94D IFFT

4D Light Field

9*9=81 spectral copies

200*200*9*9

A Theory of Mask-Enhanced CameraA Theory of Mask-Enhanced Camera

Main LensObject Mask Sensor

•Mask == Light Field Modulator

•Intensity of ray gets multiplied by Mask

•Convolution in Frequency domain

Band-limited Light Field

Sensor Slice – Fourier Slice Theorem

Photo = Slice of Light Field in Fourier Domain

[Ren Ng, SIGGRAPH 2005]

How to Capture 2D Light Field with 1D Sensor ?

Band-limited Light Field

Sensor Slice

Fourier Light Field Space

Extra sensor bandwidth cannot capture extra dimension of the light field

Sensor Slice

Extra sensor bandwidth

??????

??? ???

Solution: Modulation Theorem

Make spectral copies of 2D light field

Modulation Function

Modulated Light Field

Modulation Function

Sensor Slice captures entire Light Field

Demodulation to recover Light Field

Reshape 1D Fourier Transform into 2D

1D Fourier Transform of Sensor Signal

Modulation Function == Sum of Impulses

Physical Mask = Sum of Cosines

Mask Tile

Cosine Mask Used

Where to place the Mask?

Sensor

Mask Modulation Function Mask Modulation

Function

Mask Sensor

Where to place the Mask?

Mask Modulation Functionfx

Captured 2D Photo

Encoding due to Cosine Mask

Computing 4D Light Field2D Sensor Photo, 1800*1800 2D Fourier Transform

2D FFT

Rearrange 2D tiles into 4D planes200*200*9*94D IFFT

4D Light Field

9*9=81 spectral copies

200*200*9*9

Digital Refocusing

Only cone in focus

Captured Photo

Full resolution 2D image of Focused Scene Parts

Captured 2D Photo

Image of White Lambertian Plane

divide

Coding and Modulation in Camera Using MasksCoding and Modulation in Camera Using MasksMask? Sensor

Mask SensorMask

Sensor

Coded Aperture for Full Resolution

Digital RefocusingHeterodyne Light

Field Camera

Discussion Topics

• What is the info content of a 3D scene?– Encoding appearance and geometric complexity

• Two approaches for multi-view capture or display, – Lenslet (multiscopic), pin-hole array (parallax barrier)

• Third choice – Multiplexing– coding: can we build a display on this principle

• Mask can go anywhere, what else can we achieve?

• Should we think about multi-camera arr like this

Mask-based Approaches

• Coded Illumination– Motion Capture [2007]

• 6D Display– Lighting aware [2008]

• Optical Heterodyning– Light Field Capture [2007]

http://raskar.info

P rojec tor

P os=0

P os=255

Discussion Topics• What is the info content of a 3D scene?

– Encoding appearance and geometric complexity

• What are the dimensions beyond viewpt?– Lighting?

• What other optical sensors we can use?

• What are other display technologies?– Materials, configuration

Raskar Banff

Technology

Transcript of Raskar Banff

Looking Back: Reflections on Banff 1991 From Banff 2003

Computational Light Transport Computational Photography ... · Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar Computational Light Transport Computational Photography

Raskar Next Billion Cameras Siggraph 2009

Computational Photography - B3 Aggregationweb.media.mit.edu/~raskar/photo/2008/2008_B3TumblinAggregation06.pdfClass: Computational Photography, Advanced Topics Debevec, Raskar and

Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/.../AllSlidesraskarCodedPhotographyACCV2007.pdf · Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform,

Raskar 2012, Idea Hexagon

WASTE AUDIT REPORT - Banff Centre Centre... · The Banff Centre Waste Audit – November 2005 WASTE AUDIT REPORT The Banff Centre Banff National Park, AB. ... Banff High School was

Raskar TED global 2012: Femto Photography

Mitsubishi Electric Research Labs Display Dilemma Raskar, Oct 03 The Large Display Dilemma Ramesh Raskar Mitsubishi Electric Research Labs, Cambridge,

Raskar Graphics Interface May05 Web

Raskar 6Sight Keynote Talk Nov09

Mitsubishi Electric Research Laboratories Raskar Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA, USA Pocket Projectors All-in-one.

Introduction to Camera Challenges - Ramesh Raskar

Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005 Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric.

2016-2017 - Banff Canmore Community Foundation€¦ · of Banff, Fairmont Banff Springs Hotel, Banff Centre, Banff & Lake Louise Tourism and the Banff Canmore Community Foundation.

Raskar Ilp Oct08 Web

Raskar Coded Opto Charlotte

The Banff Centre BANFF MOUNTAIN FILM FESTIVAL WORLD TOURcenter.shiretoko.or.jp/wp/wp-content/themes/center/img/renewal/fes.… · The Banff Centre BANFF MOUNTAIN FILM FESTIVAL WORLD

Banff Buzz

Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar MIT Media Lab Ramesh Raskar Associate Professor.