Mobile 3D Scanning and Applications Prof. Dr. Marc Pollefeys ETH Zurich

17 March, 2015 http://www.inf.ethz.ch/personal/marc.pollefeys/

3D Scanning

Time-of-flight

X

d =c

2Dt

Mesa Imaging’s SwissRanger

new MS Kinect

PMDTec

3D -> 2D image projection

C X

x L

2D -> 3D ?

C x

l’

X?

x’

X

C’

L’ Triangulation

L

Structured light

P x

l’ x’

X

C’

L’

L

Replace one camera with projector/known laser pattern

PrimeSense

Mantis Vision

(Pollefeys et al. ICCV98, VSMM98, 3DIM99, IJCV04, ...)

Turning 2D street-side video into 3D models Street-Side Video Real-Time Stereo (Pollefeys IJCV08;

Gallup DAGM10; …)

real-time processing of video (30fps on PC, leveraging GPU)

…

Joint 3D reconstruction and class segmentation

joint reconstruction and segmentation (ground, building, vegetation, stuff)

(Haene et al CVPR13, …)

Building Ground Vegetation Clutter

reconstruction only (isotropic smoothness prior)

(Lee et al CVPR13; Heng et al. ICRA14; Haene et al…)

cameras as main sensors: no lasers, no GPS omnidirectional visual simultaneous localization and mapping

3D mapping for autonomous driving robust image-based localization

depth from motion stereo live obstacle maps

Computer Games

Body tracking

S

S1 S2

regression tree

Randomized forest (Shotton et al.)

based on simple depth difference tests at each node

Raw Kinect output: Color + Depth = RGBD

http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect

3D Video with Kinect

Kinect-based 3D avatar scanning

CVPR 2014 demo (Medioni et al. USC)

Sense 3D Scanner

Home Scanning

(now Apple)

Tablet + Depth Sensor

Structure Sensor

Tablet + Depth Sensor

Low power (<1W) models for tablet and for PC

depth from 2 (or 1) cameras + (un)structured laser pattern

ETH Zurich as Project Tango partner

Some of our contributions…

Today‘s mobile phones

Mobile phones are great for computational photography

• High-resolution digital camera

• Computing power (CPU+GPU+...)

• Additional sensors (accelerometers, gyroscopes, ...)

• Storage

3D reconstruction services in the cloud

Catch http://www.arc3d.be/

Recording cultural heritage

(Delaunoy et al. CVPR14)

3D Scan and Print

3D Portraits

The Selfie Authentication

Company®

+ =

• Use of individual smartphones is the only way to

massively conquer the identity management market

• Smartphones now embed enough CPU power for 3D face

computation

• 3D live-face identification extremely hard to hack (no

picture …)

Metaio’s ikea catalogue AR

captured with mobile app

Towards 3D capture + augmented reality

Microsoft HoloLens

Future?

Fully autonomous vision-based MAV

sensing: video cameras + IMU full on-board processing indoor and outdoor operation obstacle avoidance, mapping and exploration no laser, no GPS, no network

More on PixHawk: http://pixhawk.ethz.ch

(Fraundorfer et al. IROS12 best paper finalist)

(Meier ICRA11, ICRA15,...)

also Open Source software and hardware initiative

Interactive Navigation of casually captured videos

Collection of videos of the same event from different angles

Flickr/Paul J Coles

Navigation in space and time

Interactive Navigation of casually captured videos

4D Video

Interactive viewer, more results & datasets available at: http://cvg.ethz.ch/research/unstructured-vbr/

Conclusion

• 3D scanning capabilities will become ubiquitous

• Full 6DOF localization of devices in environment

– Games

– Navigation

– Augmented reality

• Will enable tight interaction between physical and digital world

Live demos @ Swiss Pavilion (Hall 9, Stand A28)

Thank you!