Augmented  reality  techniques

Sergi  Grau  Carrión  

http://sergigrau.cat  

@sergi_grau

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png

Adapted  milgram's  VR-­‐AR  continuum

Virtual  reality  -­‐  Immersive

http://www.youtube.com/watch?v=yf0sllpZx3w

Virtual  reality  -­‐  Immersive

http://www.youtube.com/watch?v=WYG7xGFByJ8

Virtual  reality  -­‐  Semi  immersive

http://www.youtube.com/watch?v=7MVyT1lgGGQ

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png

Adapted  milgram's  VR-­‐AR  continuum

Augmented  reality

Started  in  1957  (Roosevelt  home)

Augmented  reality

http://www.youtube.com/watch?v=DjevdT5XpFw

Augmented  reality

Augmented  reality

Augmented  reality

Augmented  reality

See-­‐through

Augmented  reality

Monitor  based

Augmented  reality

Augmented  reality

Augmented  reality

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png

Adapted  milgram's  VR-­‐AR  continuum

TUI

TUI

http://www.ted.com/talks/david_merrill_demos_siftables_the_smart_blocks.html

TUI

https://vimeo.com/82277562

http://en.wikipedia.org/wiki/File:Adapted_milgrams_VR-­‐AR_continuum.png

Adapted  Virtuality-­‐Reality  continuum

Augmented  Reality

http://wearables.unisa.edu.au

Capture  real  world  Computer  Vision

Capture  virtual  world  Computer  Graphics

Diminished  reality

http://www.youtube.com/watch?v=FgTq-­‐AgYlTE

Diminished  reality

http://tangible.media.mit.edu/project/focalspace/

Ingredients  for  successful  AR  

http://www.youtube.com/watch?v=yDjHbLjtL-­‐c

Environment  analysis

Realistic  augmentation+

Environment  analysis

• Geolocalization  • Electro-­‐magnetic  • Acoustic  • Inertial  • Vision-­‐based  (active  or  passive)

Real-­‐time  3D  tracking  of  user  viewpoint  w.r.t.  World

Image-­‐based  3D  tracking

Inside-­‐out

Image-­‐based  3D  tracking

Outside-­‐in

OpenCV

http://www.youtube.com/watch?v=Z43_hCM74rU

Color  tracking

Hand  tracking

http://www.youtube.com/watch?v=B4dwu3si9x0

Hand  Tracking

http://www.youtube.com/watch?v=CYJjQgV0mZY

Optical  Flow

http://www.youtube.com/watch?v=5rR_9YIcg_s

Optical  Flow

Optical  Flow

http://www.youtube.com/watch?v=1D93RmW_eN4

Optical  Flow

http://www.youtube.com/watch?v=C95bngCOv9Q

Optical  Flow

http://www.youtube.com/watch?v=0L34HsXXU00

Face  tracking

http://www.youtube.com/watch?v=JyBMxeVCQkc

Head  tracking

Face  tracking

http://www.youtube.com/watch?v=h9kPI7_vhAU

Face  tracking

Face  tracking

Face  substitution

http://auduno.github.io/clmtrackr/examples/facesubstitution.html

Image-­‐based  3D  reconstruction

Computer  vision  is  good  at  location  features:  Points  Regions  Textures  Contours  

...

Image-­‐based  3D  reconstruction

Geometry

3D  reconstruction:  what  is  where?

Semantic:  augmentation  of  context  Object  recognition  

Interpretation  of  content

3D  reconstruction  of  regions

Based  on  planar  equations.  – Hypothesis:  world  is  piece-­‐wise  planar

3D  registration  by  Iterative  Closest  Point  (ICP)

ARToolkit

http://www.youtube.com/watch?v=VR12e7WnkxU

ARToolkit

http://www.youtube.com/watch?v=YWG4nhdsySE

ARToolkit

ARToolkit

Fast  and  cheap  6D  marker  tracking  Easy  calibration  routine:  

–Measure  camera  properties:  • Center  point  of  the  camera  image.  • The  lens  distortion.  • Camera  focal  length.  !!!

Virtual  objects  appear  only  when  complete  markers  are  visible.

Image  processing

Image  processing

Image  processing

Image  processing

Image  processing

Pattern  matching

Feature  tracking:  SIFT

Markerless  tracking

Depth  cameras

Infer  depth  for  ALL  pixels

Depth  cameras

Zcam  (3DV  Systems)  2007  –  1.3Megapixel  60fps

ZCam

Kinect

http://www.youtube.com/watch?v=YTBvjLGDluY

Kinect

http://www.youtube.com/watch?v=NqjopQmqWAE

Kinect

Kinect

Kinect

Motion  capture

http://people.csail.mit.edu/rywang/handtracking/handtracking.mov

Motion  capture

Markless  motion  capture

http://vimeo.com/33323400

http://structure.io

http://structure.io

Depth  Cameras

Solves  the  “where”  but  not  the  “what”:    objects  must  still  be  identified  (segmented)

Realistic  augmentation

Environment  analysis

Realistic  augmentation+

!2  components:       Geometric  coherency       Lighting  coherency    

Realistic  augmentation

CG  Blending

Approximate  real  world  to  simulate  interaction

Virtual  in  front  of  real

Virtual  behind  real

Light  coherency

http://www.kevinkarsch.com/publications/sa11.html

Light  coherency

Light  coherence

http://www.youtube.com/watch?v=P3roYysI6pM

Light  probe

Aim:  Capture  light  coming  from  light  sources  in  real  world

Light  probe

Light  probe

Half-­‐life  2

Light  coherence

http://www.youtube.com/watch?v=ihhKLex2Na8

Light  coherence

http://handheldar.icg.tugraz.at/

Camera  effects

AR  Hardware

LG  Lifeband  Touch

Sony  SmartBandEdison-­‐  Intel‘s  Development  BoardHeapsylon  Sensoria  Fitness  socks

Lumus  DK-­‐40  AR  Glasses Oculus  Rift  ‘Crystal  Cove’ Sony’s  ‘wearable  HDTV’

Pebble

Google  Glasses

http://optinvent.com/

ORA  Glasses

Epson  Moverio  BT-­‐100

Meta  Pro  Glasses

Lumus  HUD  Display

Telepathy  One

NTT  DOCOMO

http://www.reconinstruments.com

vrAse

Reevu  Intelligent  Helmet’s  heads  up  display

Intel  RealSense  3D  cameraCreative  Senz3D

SeeSpace  InAir

 i-­‐Air  Touch  Technology

Interaction  and  Collaboration

http://www.youtube.com/watch?v=RlHOhYF_jqM

Interaction  and  Collaboration

http://www.youtube.com/watch?v=Y3TzhXYF0Lg

Augmented  reality  techniquesSergi  Grau  Carrión  

http://sergigrau.cat  

@sergi_grau