High-quality Scanning usingTime-Of-Flight Depth Superresolution

17nd March 2008

Sebastian Schuonschuon@cs.stanford.edu

Prepared for: Final Presentation CS223B, Winter 2008

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 2

Problem StatementTime-of-Flight (TOF) Cameras Have a Low Resolution

ZCam by 3DV Systems (Israel)

Measurement principle: time of flight

Depth recording: 320x240, 8bit

► Less noisy depth images with higher resolution desired

Native resolution Superresolution (4x) Geometry renderingfrom superresolution

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 3

ApproachCombine Several Images to Increase Resolution

Use multiple, here N=15, recordings from different viewpoints by translating the camera

Estimating the high resolution image resembles to an optimization problem

Optimization is multi-objective: similarity and smoothness is enforced

►

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 4

ResultsSubtle Details Become Visible and Noise is Reduced

De

pth

ma

p3

D g

eo

me

try

ren

de

ring

Native resolution Superresolution (4x)

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 5

Backup

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 6

Hidden SlideDistribution of Work

Project solely undertaken by Sebastian Schuon (me)

Supervision / Collaboration:Christian TheobaltJames Davis (UCSC)

Additional imagery for paper provided byHylke Buisman

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 7

SuperresolutionGoals

Enhance resolution

Reduce noise

Recording Resolution

(320 x 240)

(Contrast enhanced)

(Contrast enhanced)

Super Resolution(4x upsample)

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 8

Depth CameraTheory

ZCam by 3DV Systems (Israel)

RGB and Depth camera in one housing (“RGBD”)

RGB: 640x320 @ 30fps, Depth: 320x240, 160x120 @ 30fps

Measurement principle: time of flight

Depth image: distance between camera and object (not Z-coordinate)

Unprojection to 3D coordinates necessary

Specification of tracking window required

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 9

Depth CameraResults

Recording scene with different layers of depth

Black, shiny surfaces tend to be problematic

One needs to know where to record (tracking window)

Unprojection of depth images leads to 3D representation

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 10

Depth CameraNoise Characteristic

Outer regions tend to be a lot more noisy

Noise can be approximated with Gaussian

Hypothesis: Noise increases quadratic with distance

Hypothesis : Noise is correlated with color of object recorded

Variance Plot Pixel Distribution over Time at Center

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 11

Depth CameraSystematic Bias

RBG Processing disabled

Variance Plot (RGB disabled)

RBG Processing enabled

Variance Plot (RGB enabled)

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 12

SuperresolutionTheory

Based on Shift-And-Add family of algorithms

Quite well studied for grayscale and color images, overview in [Farsiu04]

We used Bilateral Shift-And-Add [Farsiu03]

Formulation as inverse problem

Our approach: rotating camera

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 13

SuperresolutionResults Simple Approach

Depth Image - Superresolution(Contrast enhanced)

Depth Image - Recording Resolution(Contrast enhanced)

3D Rendering - Recording Resolution

3D Rendering - Superresolution

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 14

Conclusion

Findings

Camera Interface and Software still beta / undocumented

Interesting effects can happen, that are not expected

Superresolution on depth images is feasible

Further steps

Reimplementation of known algorithms

Ideas for improvement:– Depth camera specific noise characteristic– Incorporating confidence map

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 15

Depth CameraSystematic Bias

Sebastian Schuon schuon@cs.stanford.edu Project_26.ptt 16

SuperresolutionImproved Approach