User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D...
Transcript of User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D...
![Page 1: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/1.jpg)
Mantis Vision MV4D
A bit about me
USER GENERATED 3D CONTENT CREATION
![Page 2: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/2.jpg)
Mantis Vision MV4D
Mobile Depth Sensing Methods What are they good for?
![Page 3: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/3.jpg)
Mantis Vision MV4D
FACT: 3D is here. And it’s going mobile.
![Page 4: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/4.jpg)
Mantis Vision MV4D
Apps are next. Content Creation is the key.
![Page 5: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/5.jpg)
Mantis Vision MV4D
3D Use Cases
On desk AR
Indoor navigation -> Augmentation – Modeling
Hand\body tracking\gestures
• Facial tracking – Avatar
• Human \ objects modeling and printing
Computational photography
3D videos-captured events to share in the social media
Robotics vision
New killer apps – Not on the list
![Page 6: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/6.jpg)
Mantis Vision MV4D
Each depth sensing technology has its own set of
LIMITS, BENEFITS & VALUES.
Knowing these is the key to a winning mobile experience.
I hope today’s talk can help...
![Page 7: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/7.jpg)
Mantis Vision MV4D
3D SENSING
TECHNOLOGIES
THE PLAYERS
TIME OF FLIGHT TRIANGULATION
PASSIVE ACTIVE
![Page 8: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/8.jpg)
Mantis Vision MV4D
TIME OF FLIGHT – PRINCIPLES OF OPERATION
•Time of Flight (TOF) – Direct – as its name implies (3DV) •(TOF) – Phase detection of modulated light source (PMD, SoftKinetics, KinectOne) 2.5m back and forth from an object takes 16.67ns…
![Page 9: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/9.jpg)
Mantis Vision MV4D
TIME OF FLIGHT – SAMPLE DATA SHOTS
oExplanary videos (30 sec)
oSome sample data shots:
•Fast •Robust •High-Noise
State of the Art TOF Doesn’t meet mobile
constraints (size\power\cost)
![Page 10: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/10.jpg)
Mantis Vision MV4D
TIME OF FLIGHT – PROS AND CONS
Pro:
Direct measurement per pixel
Low computation and latency
Ideal for gesture\Human interactions
Cons:
High amount of depth noise ~ centimeter scale
Biased depth results due to object reflectivity, ambient
light, edges
Absolute dimensions are not preserved well
• Phase shift calculation requires multiple, very short exposure integration within the
duration of a single modulated cycle (20-130MHz).
• This means large pixels (>10um) which limits sensor resolution.
![Page 11: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/11.jpg)
Mantis Vision MV4D
TRIANGULATION– PRINCIPLES OF OPERATION
Requires: Baseline (A) Correspondence (B) Localization (C)
![Page 12: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/12.jpg)
Mantis Vision MV4D
PASSIVE TRIANGULATION – CORRESPONDENCE CHALLENGE
• Correspondence issues are the main challenge as nothing ensures distinctive features across images.
*Images by Pelican Imaging
![Page 13: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/13.jpg)
Mantis Vision MV4D
PASSIVE TRIANGULATION METHODS
• Stereo cameras
• Multi aperture
• Shape from multiple images
• Pros: Passive • Cons: Texture dependent
• Pros: Multi-view Robustness Computational Photography (Refocus) • Cons: Texture dependent • and very small baseline = low depth
accuracies
• Pros: Large virtual baseline = High depth accuracies Use standard back camera
• Cons: Static only • Cloud only processing
Complicated capture process texture dependent
![Page 14: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/14.jpg)
Mantis Vision MV4D
SOLVING THE CORRESPONDENCE PROBLEM
Coded light source
Instead of looking for correspondence in a featureless image …
Create your own by replacing one camera with a coded light source
![Page 15: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/15.jpg)
Mantis Vision MV4D
CODED LIGHT CHALLENGE - INDEXING
How to index projected features in space ? •Time multiplexed – limited motion •Unique cluster of points – Kinect360 •Dense bi-dimensional epipolar code-
Mantis Vision
![Page 16: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/16.jpg)
Mantis Vision MV4D
Active Triangulation – Kinect360 vs. MV4D
KINECT MV4D Robust Correspondence, High Fill Factor, High Code Capacity Robust Correspondence, Low Fill Factor
![Page 17: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/17.jpg)
Mantis Vision MV4D
![Page 18: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/18.jpg)
Mantis Vision MV4D
![Page 19: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/19.jpg)
Mantis Vision MV4D
![Page 20: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/20.jpg)
Mantis Vision MV4D
![Page 21: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/21.jpg)
Mantis Vision MV4D
![Page 22: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/22.jpg)
Mantis Vision MV4D
![Page 23: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/23.jpg)
Mantis Vision MV4D
![Page 24: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/24.jpg)
Mantis Vision MV4D
![Page 25: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/25.jpg)
Mantis Vision MV4D
280 critical features in selected region…Vs.
![Page 26: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/26.jpg)
Mantis Vision MV4D
40… That’s a x7 difference
![Page 27: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/27.jpg)
Mantis Vision MV4D
Where does it matter?
![Page 28: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/28.jpg)
Mantis Vision MV4D
The noisier the depth data, the greater the need to rely on heavy
de-noising\averaging
Yan Cui, Derek Chan, Sebastian Thrun and Christian Theobalt. See http://ai.stanford.edu/~schuon/ for details.
How does TOF compare?
![Page 29: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/29.jpg)
Mantis Vision MV4D
AVERAGING HAS ITS LIMITS
o Works only for static scenes
o Reduces actual frame rate\speed for clean surfaces (x 5-20 depending on
quality of data)
o Loss of small features, as a result of amount of depth noise (STD)
![Page 30: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/30.jpg)
Mantis Vision MV4D
DEPTH NOISE VS. DISTANCE
We wanted to put both passive and active methods on same graph, but…
0
200
400
600
800
1000
1200
1400
1600
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
F5-Pro
Kinect
Pelican
![Page 31: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/31.jpg)
Mantis Vision MV4D
DEPTH NOISE (STD) VS. DISTANCE
Therefore…
0
5
10
15
20
25
30
35
40
45
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Stan
dar
d d
evi
atio
n in
mm
Distance (m)
Depth noise (mm) Vs distance (m)
F5-Pro
Kinect
![Page 32: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/32.jpg)
-- -- -- -- --
TOF –
MOBILE
PASSIVE
STEREO
SHAPE
FROM
MOTION
MULTI
APERTURE
KINECT
MV4D
NUI
INDOOR
AR
LARGE
SCALE
DESK AR
SMALL
SCALE
MODELIN
G
UGC 3D
STATIC
UGC 3D
DYNAMIC
FACIAL
TRACKIN
G
SUMMARY
TABLE
(by app and
technology)
POOR MID EXCELLENT
![Page 33: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/33.jpg)
![Page 35: User Generated 3D Content Creation | GTC2014...Content Creation is the key. Mantis Vision MV4D 3D Use Cases On desk AR Indoor navigation -> Augmentation – Modeling ... Computational](https://reader034.fdocuments.in/reader034/viewer/2022051606/602a4c66b7d276368f10c5dd/html5/thumbnails/35.jpg)