Layered 3D Tomographic Image Synthesis for Attenuation-

based Light Field and High Dynamic Range Displays

Gordon Wetzstein

Wolfgang Heidrich

Douglas Lanman

Ramesh Raskar

UBC MIT Media Lab

3

Overview

Attenuation Layers

with Spacers

Backlight

4

Key Insights

Multi-Layer Display

Computed

Tomography

4D Light Field

5

Glasses-Free 3D Display

Lenslet Arrays Parallax Barriers Layered 3D

Nin

ten

do 3

DS

Alio

sco

py 3

DH

D 4

2”

Ives 1903 Lippmann 1908

sourc

e: w

ikip

edia

6

Gotoda 2010, 2011

Generalizing Parallax Barriers

mask 1

mask 2

mask 1

mask 2

mask 2

mask 3

mask K

mask 1

… Conventional Parallax Barriers

Lanman et al. 2010

Layered 3D

Tem

pora

l M

ultip

lexin

g

Multiple Layers

We are first to analyze problem and build prototypes

7

Computed Tomography (CT)

sourc

e: w

ikip

edia

x-ray source

x-ray sensor

3D Reconstruction

Reconstructed 2D Slices

8

Tomographic Light Field Synthesis

q

2D Light Field

x

x q

Backlight

Attenuation Volume

Virtual Planes Image Formation

L(x,q ) = e- m (r )dr

cò

c

drrxL )(,log q

P)log( L2

20

P)log(argmin

L

Tomographic Synthesis

9

CT vs. Layered 3D

Computed Tomography Layered 3D

reconstruct physical volume

sensor noise

thin stack of optimized layers

no noise

10

Multi-Layer Decomposition

Input 4D Light Field

Optimized Attenuation Layers

1 2

3 4

5 1 2 3 4 5

Photographs of Prototype

viewer moves right

vie

wer m

oves d

ow

n

11

Depth of Field for 3D Displays Integral Imaging Parallax Barriers

Antialiasing + Display Prefilter Antialiasing + Display Prefilter Zwicker et al. 2006

Display Thickness

Distance of Virtual Plane from Middle of Display (cm)

Cu

toff (

cycle

s/c

m)

Maximum Resolution

12

Integral Imaging Parallax Barriers Layered 3D

?

Distance of Virtual Plane from Middle of Display (cm)

Cu

toff (

cycle

s/c

m)

Display Thickness

Maximum Resolution

How Do Layers Increase Depth of Field?

13

h u

a

backlight Layered 3D

do

ξ

Chai et al. 2000; Durand et al. 2005;

Veeraraghavan et al. 2007;

Lanman et al. 2008; Ihrke et al. 2010

Fourier Transform

u

a

Light Field

fu

fa

f0 -f0

Light Field Spectral Support

attenuator

Review of Frequency-Domain Light Field Analysis

?

14

u

a

h u

a

u

a

Layered 3D

Ä

Fourier Transform

Rear Layer Light Field Front Layer Light Field

fu

fa

f0 -f0

Rear Layer Light Field Spectral Support

fu

fa

f0 -f0

Front Layer Light Field Spectral Support

backlight

attenuator

Multi-Layer Frequency-Domain Analysis

15

u

a

h

u hb

backlight Parallax Barrier

Layered 3D

fu

fa

Emitted Light Field Spectral Support

backlight

attenuator

Multi-Layer Frequency-Domain Analysis

16

u

a

h

fξ

do

ξ

fu

fa

fξ

Layered 3D

Two-Layer Depth of Field

Two-Layer Spectral Support

backlight

attenuator

Distance of Virtual Plane from Middle of Display (cm)

Cuto

ff (

cycle

s/c

m)

Depth of Field

Multi-Layer Frequency-Domain Analysis

17

u

a

h

do

ξ

fu fu

fa fa

fξ

fξ

Layered 3D

Two-Layer Spectral Support Three-Layer Spectral Support

backlight

attenuator

fξ

Distance of Virtual Plane from Middle of Display (cm)

Cuto

ff (

cycle

s/c

m)

Depth of Field

Three-Layer Depth of Field

Multi-Layer Frequency-Domain Analysis

18

Multi-Layer Depth of Field

Conventional*

Layered 3D *Includes integral imaging and parallax barriers

Distance of Virtual Plane from Middle of Display (cm)

Cu

toff (

cycle

s/c

m)

19

Optimization: Number of Layers

Two Layers Three Layers Five Layers

20

Optimization: Display Thickness

Number of Layers

PS

NR

in d

B

Average Reconstruction PSNR for All Scenes

21

Application to HDR Display

“Square Root” Layers

22

Application to HDR Display

“Square Root” Layers

23

Application to HDR Display

Optimized Layers

25

Limitations: Field of View

FOV 10º FOV 20º FOV 45º

26

Personal Glasses-Free 3D Display

Challenges for dynamic display:

Real-time computation

Engineering issues, moiré

27

Dynamic Multi-Layer LCDs

Dynamic Light Field Display using Multi-Layered LCDs, to appear in Siggraph Asia 2011 Douglas Lanman, Gordon Wetzstein, Matthew Hirsch, Wolfgang Heidrich, Ramesh Raskar

28

Photo-stéréo-synthesis Louis Lumière 1920

Focal Stack

Layered

Transparencies 3D effect, but not correct!

Computational Displays

Computational Photography

29

www.layered3d.com

Datasets, code & videos on the website!

30

31

3D objects only inside enclosure

mostly mechanically moving parts

all depth cues

computationally expensive Holograms

inexpensive fabrication

computationally efficient

3D objects outside enclosure

no moving parts

Volumetric Displays

Glasses-Free 3D Display

Zebra Imaging MIT Holovideo Jones et al. 2007 Sony LightSpace

Layered 3D

32

Limitations: “Flip Animations”

Parallax Barriers Multi-Layer

Full Resolution

Multi-Layer

Reduced Resolution

33

Performance Assessment

Prototype Prototype Prototype

Input Input Input Expected Error Expected Error Expected Error

34

Application to HDR Display

bottom bottom

bottom bottom

top top

top top

Se

etz

en

20

04

, D

olb

y C

an

ad

a

LCD

LED Array

35

Implementation

Prototype in Front of LCD (Backlight)

Epson Stylus 2200 inkjet (300 dpi, six color primaries)

5 layers (5.7×7.6 cm),1.27 cm thickness, 10° field of view

Transparencies and Acrylic Spacers Inkjet Transparency Printer *

*not „90s, try eBay

36

Implementation: Software viewer moves right

vie

wer m

oves d

ow

n 1

2 3

4 5

1 2 3

4 5

POV-Ray: 7×7 views (512×384 pixels), 10° field of view Depth of field tuned for combined antialiasing and display prefilter

MATLAB: LSQLIN (independently for each color channel)

12 minutes on 2.4 GHz Intel Core 2 with 8 GB RAM