MarrNet: 3D Shape Reconstruction via 2.5D...
1
MarrNet: 3D Shape Reconstruction via 2.5D Sketches Jiajun Wu 1* Yifan Wang 2* Tianfan Xue 1 Xingyuan Sun 3 William T. Freeman 1,4 Joshua B. Tenenbaum 1 1 Massachusetts Institute of Technology 2 ShanghaiTech University 3 Shanghai Jiao Tong University 4 Google (* equal contributions) Single-Image 3D Shape Reconstruction Goal: Reconstructing 3D shapes with fine details from a single image Observation: Existing methods have limitations. Solution: incorporating 2.5D sketches, inspired by Marr’s vision theory Advantages • The use of 2.5D sketches disentangles the problem into object recognition and 3D shape completion, and improves performance. • The differentiable consistencies between 2.5D sketches and 3D shape enable end-to-end fine-tuning without 3D annotations. NIPS 2017 2.5D Sketches as an Intermediate Representation (a) Images (b) 2.5D Sketches (c) 3D Shape MarrNet Training Details • Pre-train the 2.5D sketch estimation network and the 3D shape estimation network separately • Fine-tune the encoder of the 3D shape estimation net on real data with the self-supervised reprojection loss Results 3D Shape (128 x 128 x 128) (a) 2.5D Sketch Estimation (b) 3D Shape Estimation (c) Reprojection Consistency …… 2.5D Sketches 2D Image normal depth silhouette Normal Ball Blurry or Noisy Output Simple Geometry Only Estimated normals Estimated depths Ground truth Direct predictions w/o 2.5D sketches Images MarrNet Images MarrNet DRC Images MarrNet DRC Images Ground truth MarrNet 3D- VAE-GAN Images Ground truth MarrNet DRC PASCAL 3D+ IKEA Multi-Class
Transcript of MarrNet: 3D Shape Reconstruction via 2.5D...
MarrNet: 3D Shape Reconstruction via 2.5D SketchesJiajun Wu1* Yifan Wang2* Tianfan Xue1 Xingyuan Sun3 William T. Freeman1,4 Joshua B. Tenenbaum1
1 Massachusetts Institute of Technology 2 ShanghaiTech University 3 Shanghai Jiao Tong University 4 Google (* equal contributions)
Single-Image 3D Shape ReconstructionGoal: Reconstructing 3D shapes with fine details from a single imageObservation: Existing methods have limitations.
Solution: incorporating 2.5D sketches, inspired by Marr’s vision theoryAdvantages• The use of 2.5D sketches disentangles the problem into object
recognition and 3D shape completion, and improves performance.• The differentiable consistencies between 2.5D sketches and 3D
shape enable end-to-end fine-tuning without 3D annotations.
NIPS 2017
2.5D Sketches as an Intermediate Representation
(a) Images (b) 2.5D Sketches (c) 3D Shape
MarrNet
Training Details• Pre-train the 2.5D sketch estimation network and the 3D shape estimation network separately• Fine-tune the encoder of the 3D shape estimation net on real data with the self-supervised reprojection loss
Results
3D Shape(128 x 128 x 128)
(a) 2.5D Sketch Estimation (b) 3D Shape Estimation
(c) Reprojection Consistency
……
2.5D Sketches
2D Image
normal
depth
silhouette
Normal Ball
Keyp
oint
5IK
EASU
N
Input Before FT After FT Input Before FT After FT Input Before FT After FT
Keyp
oint
5IK
EASU
N
Input Before FT After FT Input Before FT After FT Input Before FT After FT
Blurry or Noisy Output Simple Geometry Only
Estimated normals
Estimated depths
Ground truthDirect predictions w/o 2.5D sketches
Images MarrNetImages MarrNetDRC Images MarrNetDRC
Images Ground truth
MarrNet3D-VAE-GAN
Images Ground truth MarrNetDRC
PASCAL 3D+ IKEA
Multi-Class
