1. CompressiveThis slide has a 16:9 media windowLight Field DisplaysGordon Wetzstein - MIT Media Lab Collaborators: Doug Lanman, Matt Hirsch, Ramesh Raskar, Wolfgang Heidrich

2. This slide has a 16:9 media window 3. viewer moves rightThis slide has a 16:9 media windowviewer moves down4D Light Field 4. Compressive Displays ComputationalDisplay OpticsProcessing 5. This slide has a 16:9 media windowDisplay-adaptiveCompression Computed TomographyNonnegative Tensor4D Light Field Factorization Compressive Optics Uniform or Directional Backlight Stacked Layers(LCDs or Transparencies) 6. Prototype Layered 3D, SIGGRAPH 2011 Attenuation Layers with Spacers Backlight 7. Video clip 8. Prototype Tensor Display, SIGGRAPH 2012 9. Video clip 10. What do we mean by glasses-free 3D?binocular disparityconvergence motion parallax accommodation/blur current glasses-based (stereoscopic) displays near-term glasses-free (light field) displays longer-term holographic displays 11. Is glasses-free 3D Technology ready? Nintendo 3DS MasterImage 3DAsus Eee Pad MeMO 3D LG Optimus 3DE3 2010 Computex 2011 Computex 2011Mobile World Congress 2011 Toshiba 3DTV PrototypeSony 3DTV Prototype LG 3DTV Prototype CES 2011 CES 2011 CES 2011 12. Parallax Barriers Ives 1903barrier2D display Low resolution & very dim Switchable 2D/3D with LCDs 13. Parallax Barriers Ives 1903barrier Nintendo 3DS2D display Low resolution & very dim Switchable 2D/3D with LCDs 14. lenslets Integral Imaging Lippmann 1908 2D display Brighter than parallax barriers Always low resolution, even for 2D 15. lenslets Integral Imaging Lippmann 1908 2D displayAlioscopy 3DHD 42(1920x1200, 1x8 views) Brighter than parallax barriers Always low resolution, even for 2D 16. Directional Backlighting 3M & MS Wedge3M Directional Backlight FilmNelson and Brott, 2010 US Patent 7,847,869LEDthin light guide LED Requires 120 Hz for stereo Not practical for multiview Microsoft Wedge 17. Directional Backlighting Lenslet array 18. Glasses-Free 3D Display LightSpace Sony Jones et al. 2007 Zebra Imaging MIT HolovideoHolograms DisplaysVolumetric 3Ddepth cues inside enclosure all objects only opticallymechanically moving parts mostly & computationally expensive 3D objects outside enclosure inexpensive off-the-shelf partsCompressive LF Displays no moving parts efficient computationally 19. From Conventional to Compressive 3D Displaysmask 2mask 1 Conventional Parallax Barriers t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 20. From Conventional to Compressive 3D Displays mask 2 mask 1Time-shifted Parallax Barriers [Kim et al. 2007]High Resolution through High Speed tt tParallax Barriers Time-ShiftedHR3DLayered 3DTensor Displays1903Parallax Barriers 2007SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 21. From Conventional to Compressive 3D Displays Perceptual Integration timeTime-shifted Parallax Barriers [Kim et al. 2007]High Resolution through High Speed t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 22. From Conventional to Compressive 3D Displays mask 2 mask 1 High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010]Compression in Time Nonnegative Matrix Factorization t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 23. From Conventional to Compressive 3D Displays Perceptual Integrationtime High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010]Compression in Time Nonnegative Matrix Factorization t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 24. From Conventional to Compressive 3D Displays mask K mask 2 mask 1Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth Computed Tomography tt tParallax Barriers Time-ShiftedHR3DLayered 3DTensor Displays1903Parallax Barriers 2007SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 25. From Conventional to Compressive 3D Displays mask K mask 2 mask 1Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth Computed Tomography tt tParallax Barriers Time-ShiftedHR3DLayered 3DTensor Displays1903Parallax Barriers 2007SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 26. From Conventional to Compressive 3D Displays mask K mask 2 mask 1Layered 3D [Wetzstein et al., SIGGRAPH 2011]Compression in Pixels Computed Tomography tt tParallax Barriers Time-ShiftedHR3DLayered 3DTensor Displays1903Parallax Barriers 2007SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 27. From Conventional to Compressive 3D DisplaysPerceptual Integration time Tensor DisplaysCompression in Time & Pixels Tensor Factorization t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 28. From Conventional to Compressive 3D DisplaysPerceptual Integration timeTensor Displays Multilayer & Directional Backlighting t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 29. From Conventional to Compressive 3D DisplaysPerceptual Integrationthin!timeTensor Displays Directional Backlighting t t tParallax Barriers Time-Shifted HR3DLayered 3DTensor Displays1903Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012 30. Prototype Layered 3D, SIGGRAPH 2011 Attenuation Layers with Spacers Backlight 31. Computed Tomography (CT)x-ray sensorsource: wikipedia3D Reconstructionx-ray sourceReconstructed 2D Slices35 32. Tomographic Light Field SynthesisImage FormationVirtual Planes - c m (r )drxL(x, q ) = e Attenuation Volume log L x, (r )dr cBacklight Tomographic Synthesis 2D Light Field log( L )P2argmin log( L) P20x 36 33. CT vs. Layered 3DComputed Tomography Layered 3D reconstruct physical volume thin stack of optimized layers sensor noise no noise 37 34. Multi-Layer Decompositionviewer moves rightviewer moves downInput 4D Light Field1 2 3 1 2 4 3 5 4 5Photographs of PrototypeOptimized Attenuation Layers 35. Depth of Field for 3D DisplaysIntegral ImagingParallax BarriersCutoff (cycles/cm) Maximum ResolutionDisplay Thickness Zwicker et al. 2006 Antialiasing + Display Prefilter Distance of Virtual Plane from Middle of Display (cm) 36. How Do Layers Increase Depth of Field?Integral ImagingParallax BarriersLayered 3DCutoff (cycles/cm) ?Maximum ResolutionDisplay Thickness Distance of Virtual Plane from Middle of Display (cm) 37. Optimization: Number of LayersTwo Layers Three LayersFive Layers 38. Optimization: Display Thickness Average Reconstruction PSNR for All ScenesPSNR in dB Number of Layers 39. Application to HDR Display Square Root Layers 40. Application to HDR Display Square Root Layers 41. Application to HDR Display Optimized Layers 42. Video clip 43. Limitations: Field of ViewFOV 10 FOV 20FOV 45 44. Personal Glasses-Free 3D DisplayChallenges for dynamic display: Real-time computation Engineering issues, moir 45. Multi-Layer LCD SIGGRAPH ASIA 2011 46. Barco E-2320 PAGrayscale IPS LCD1600x1200 @ 60 Hz 47. Four Stacked Liquid Crystal Panels Two Crossed Polarizers 48. Overview of LCDsIvertical polarizer color filter arrayliquid crystal cells horizontal polarizerI0backlight Malus Law Intensity Modulation with Liquid Crystal CellsI = I0 sin2 (q ) 49. Extending Layered 3D to Multi-layer LCDs Virtual Planes Design OptimizationLCD Eliminate redundant polarizers3 Sequentially-crossed designLCDx2LCD1 backlight2D Light Field x 50. Extending Layered 3D to Multi-layer LCDs Virtual Planes Design OptimizationLCD Eliminate redundant polarizers3 Use sequentially-crossedLCDx Exploit field-sequential color2 0.33 = 2.7% brightnessLCD1 backlight2D Light Field x 51. Polarization Field DisplaysVirtual PlanesDesign OptimizationLCD Eliminate redundant polarizers3 Use sequentially-crossedLCD x Exploit field-sequential color2 0.33 = 2.7% brightnessLCD Further optimize polarizers1 Minimum is a crossed pairbacklight 2D Light Fieldx 52. Modeling and Synthesizing Polarization FieldsVirtual PlanesLCDf33Image Formation f2 KLCDx Q(x, q ) = fk (x, q )2 k=1f1L(x, q ) = sin 2 (Q(x,q ))LCD1backlightTomographic Synthesis Q(x,q ) = sin-1 ( )L(x, q ) mod p 2D Light Field Q = Pfargmin Q - Pf22fmin f fmax x 53. Decompositions & Reconstructionsviewer moves rightviewer moves downStacked Polarization Input 4D Light Field Rotating Layers 90 Optimized Rotation Angles for Each Layer 0 54. Decompositions & Reconstructionsviewer moves rightviewer moves down Input 4D Light FieldReconstruction Results 55. Multi-layer LCDAttenuation Layers Polarization-Rotating Layers 56. Video clip 57. Tomographic Image Synthesis Target Light Field Projection Matrix LCD Pixel Values= *b=Ax 58. SART - Simultaneous Algebraic Reconstruction Technique b=Ax pre-compute some weightsinitial guessATv Axupdateclamp 59. Implementing Ax as Multiview Rendering 60. Implementing ATv as Projective Texture Mapping 61. Benefits & Limitations 62. Light Field Slice RepresentationLight Fieldmoving to the right 63. Light Field Slice Representation Light FieldLight Field Slice moving to the left 64. Light Field Slice RepresentationLight FieldView from AboveLight Field Slicemoving to the left 65. Light Field Slice Representation Light FieldMultilayer Light Field DisplayLight Field Slice moving to the leftFront LayerMiddle LayerRear LayerBacklight 66. Light Field Slice RepresentationLight FieldMultilayer Light Field Display Light Field SliceL(moving to the leftFront Layer fm(3)(Middle Layer fm(2)(L(Rear Layer fm(1)(Backlight 67. Light Field Tensor RepresentationLight FieldMultilayer Light Field Display Light Field TensorL(Front LayerRear Layer fm(3)(Middle LayerL( fm(2)(Rear Layer fm(1)(Backlight 68. Light Field Tensor RepresentationLight FieldMultilayer Light Field Display Light Field Tensor L(Front LayerRear Layer fm(3)(Middle LayerL( fm(2)(Rear Layer fm(1)(Backlight 69. Light Field Tensor RepresentationLight FieldMultilayer Light Field Display Light Field TensorFront LayerRear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight 70. Light Field Tensor RepresentationLight FieldMultilayer Light Field Display Light Field TensorFront LayerRear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight 71. Light Field Tensor RepresentationLight FieldMultilayer Light Field Display Light Field TensorFront LayerRear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight 72. Light Field Tensor DecompositionTarget Light Field Tensor Rank-M Approximation Nonnegative Tensor Perceptual Factorization (NTF)Integration+ + ... +Frame 1 Frame 2 Frame M 73. Light Field Tensor DecompositionTarget Light Field Tensor Rank-M Approximation Nonnegative Tensor Perceptual Factorization (NTF)Integration+ + ... +Frame 1 Frame 2 Frame M 74. Light Field Tensor Decomposition Nonlinear (Multilinear)Optimization Problem Iterative Update Rules (see paper for details) Efficient GPU ImplementationForward Projection (Multiview Rendering) Back Projection (Projective Texture Mapping) 75. Design Tradespace: Layers vs. Frames PSNR without Directional Backlight# layers# frames PSNR with Directional Backlight# layers# frames 76. Design Tradespace: Layers vs. FramesPSNR without Directional Backlight# layers# frames PSNR with Directional Backlight# layers# frames 2 Layers, Layers,DirectionalBacklight (Tensor Display)3 Layers, 1 Frame Frames 3D SIGGRAPH 2010) 1 Layer, 3 3 3 Frames(Layered (Tensor Display) 2011)Frames, 3 (HR3D SIGGRAPH Asia Original Original2 Layers, 3 Frames 3 Layers, 1 Frame3 Layers, 3 Frames 1 L, 3 F, Directional BL 77. Tensor Display Prototypes Reconfigurable Directional BacklightThree Layer 78. Tensor Display Prototypes3 LayerLCDDirectionalBacklightHardware 79. Three Layer PrototypeVideo clip 80. Video clip 81. Directional Backlight Prototype Video clip 82. LCD + Directional BLView from above LCD with Directional Backlight, Rank 6Directional BL 83. LCD with Directional Backlight, Rank 6 (as seen by obserer)Video clip 84. LCD with Directional Backlight, Rank 6 (as seen by obserer)Video clip 85. Filmed with High-speed Camera Directional BacklightVideo clipFront LCD 86. Video clipLenslets only LCD+DBL Rank 1 LCD+DBL Rank 6 87. Limitation: Compressible Light FieldsNatural 4D Light Field Random 4D Light Field 88. Next-generation Technology What about Content?Computational PhotographyConsumer Light Field CamerasRendered Footage Computational Displays Camera Rigs 89. SIGGRAPH 2012 Course onComputational DisplaysCode & Datasets onlineUse Layered 3D in your class!media.mit.edu/~gordonwcameraculture.media.mit.edu