Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented...

34
Introduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis

Transcript of Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented...

Page 1: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Introduction to Augmented

RealityCreating Augmented reality costume: concepts, tools, decisions, failures

Aurelijus Banelis

Page 3: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Augmented Reality

Page 4: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

What is

To the end-user

Augmented Reality

Page 5: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

● Real world

Page 6: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

● Real world● Camera

Page 7: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

● Real world● Camera● Artificial

Page 8: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

● Real world● Camera● Artificial● Follows

Page 9: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

AugmentedReality

Page 10: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Hello worldexample

Complete product

Page 11: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis
Page 12: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Augmented Reality

Computer Vision

3D Graphics

Infrastructure

Page 13: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

Page 14: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

Page 15: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Page 16: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Page 17: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

RAW data → ObjectsData from

cameraNoise,Useful

Analyse,Compare

Meaning,decision

125142365814152451542247582451424125384864535315315313547876815453151313515748448343818131

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Found { marker=”m12” x = 10, y = 20, accuracy=0.8}

Found { marker=”m98” x = 15, y = 44, accuracy=0.6}

001000011100010010100001110010000001010110101001001000101111101010010000101101001000101000

Page 18: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

Noises by nature

Page 19: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

So filters in use

Page 20: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

Including redundancy

Folding

if m

arke

r1 a

nd m

arke

r2Low light

Low resolution

Similarity

Page 21: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

cameraData from ARToolkit

3D models to be drawn

⇒ ++

Page 22: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

3D models to be drawn

⇒ ++

Page 23: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

Found { marker=”m12” rot = (10, 20, 5) scale = 0.3 pos = (5, 4, 22)}

Found { marker=”m98” rot = (5, 33, 44) scale = 0.4 pos = (22, 8, 3)}

3D models to be drawn

⇒ ++

Page 24: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

Found { marker=”m12” rot = (10, 20, 5) scale = 0.3 pos = (5, 4, 22)}

Found { marker=”m98” rot = (5, 33, 44) scale = 0.4 pos = (22, 8, 3)}

3D models to be drawn

OpenGL ES 1.1

⇒ ++

Page 25: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

CV → 3DCombinedFrame from

camera

Captured preview image from device’s camera

Data from ARToolkit

Found { marker=”m12” rot = (10, 20, 5) scale = 0.3 pos = (5, 4, 22)}

Found { marker=”m98” rot = (5, 33, 44) scale = 0.4 pos = (22, 8, 3)}

● Frame

● Transform● Draw 3D

3D models to be drawn

OpenGL ES 1.1

⇒ ++

Page 26: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

Page 27: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

⇒ 3D Modeling soft Android

App

Idea → 3D models

Idea / 2D pictures

⇒ .objWavefront

Page 28: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

Page 29: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

Page 30: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

3D G

raph

ics

Infr

astr

uctu

re

Idea → 3D models

Normals Debug

3D Modeling soft AndroidApp

Idea / 2D pictures

⇒ .objWavefront

Page 31: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Com

pute

r Vis

ion

Infr

astr

uctu

re3D

Gra

phic

s

Invest in 3D rendering

Debug separately

Consider native implementation

Copying ARToolKit native libraries takes time

Implement custom widgets for debugging(faster than reloading whole code)

Garbage collection between Java native and Buffer structures takes time (slow loading)

Page 32: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

3D G

raph

ics

Infr

astr

uctu

reC

ompu

ter V

isio

n End-to-end

Android (SDK)

Blender

ARToolKit

QR + Server

Draw on fabric

Wraps computer vision, starting point

Custom logic (e.g. helmet position)

Generating 3D models

Telling, how to find application

Some shops requires vector images

Page 33: Introduction to Augmented Reality - FOSDEMIntroduction to Augmented Reality Creating Augmented reality costume: concepts, tools, decisions, failures Aurelijus Banelis. Aurelijus Banelis

Questions?