01/07/2010 15:36 1

Pieter Jonker Professor of Vision based Robotics

Dept Bio-Mechanical Engineering, Faculty 3MEDelft University of Technology, Delft

The Netherlandsp.p.jonker@tudelft.nl

Augmented Reality

Yolande KolsteeLector of Innovative Visualisation Techniques for Art & Design

Royal Academy of Art, The HagueThe NetherlandsY.Kolstee@kabk.nl

01/07/2010 15:36 2

Het volgen van een enkele hypothese (toegepast in AR)Augmented Reality vs Virtual Reality

01/07/2010 15:36 3

Sorts of Augmented Reality

Screen BasedHead-Set Based

01/07/2010 15:36 4

Sorts of Augmented Reality

Video see through AR2 cameras + 2 displays + video in video

Optical see through AR1 camera + 2 half translucent mirrors + 2 displays

01/07/2010 15:36 5

History of our headsets

01/07/2010 15:36 6

The Measurement Problem

How to hold the virtual objects in a perfect position and orientation in the physical world if I move my head?

By searching and tracking (camera + image processing) one or more known landmarks!

01/07/2010 15:36 7

How to anchor onto a marker?

Grab an image Edge detectionSelect closed

contoursSimple corner

detection

Keep contours with 4 corners

Split contour into 4 lines

Sub-pixel fit of the 4 lines

Intersect 4 lines for corners

Calculate pose of marker

Determine the ID

Calculate pose of camera

Grab an image Edge detectionSelect closed

contoursSimple corner

detection

Keep contours with 4 corners

Split contour into 4 lines

Sub-pixel fit of the 4 lines

Intersect 4 lines for corners

Calculate pose of marker

Determine the ID

Calculate pose of camera

1

32

1024

2

64

2048

4

128

4096

8

256

8192

512

16384

161

32

1024

2

64

2048

4

128

4096

8

256

8192

512

16384

16

01/07/2010 15:36 8

Why not tracking at natural features?

This costs too much time fort now, but soon…

01/07/2010 15:36 9

Is only a camera & image processing enough?

No, we must know where the head will be in 30 msecsbecause rendering a virtual image costs time. We have to predict where the head will be bij measuring the motion of the head.

That is why we have to fuse the data of the camera and image processing with the date from a so called inertia tracker to track fast head movements.This is comparable with the yes and the inner ear / vestibular organ of a human.

An inertia tracker has 3 accelerometers to measure linear acceleration in 3 dimensions (x, y, z) and 3 gyroscopes to measure 3 rotational velocities (roll, pitch, yaw).This gives relative displacements at 100Hz that are combined with the 25Hz absolute position and orientation of the camera.

01/07/2010 15:36 10

Fusion of camera and inertia tracker data

100 Hz

60 Hz

Camera measurement

Inertia measurement

Data Fusion

Rendering Imagetime

Image processingImage acquisition 15 Hz

01/07/2010 15:36 11

So how accurate can one place virtual objects in the physical world?

Statical misalignment< 0.6 ° (< 0.03° = eye) Dynamical Misalignment< 2.3° (< 0.5° = eye)

With a 20 x 20 cm marker at 5 m and a 1MPix camera

Own and object motion will camouflage this

01/07/2010 15:36 12

Future elektronics to loose the backpack and headset

01/07/2010 15:36 13

Conclusions so far

• Augmented Reality is not Virtual Reality• Augmented Reality is screen or headset based• Headset based AR is video or optical see-through• The AR problem is dominated by the the allignment

problem of virtual and real world objects• This is solved by inertia and camera tracking• Due to the imperfect technology one can still see a

bit alignment errors, but far better than GPS! • Happily the human visual system compensates a bit• Future hardware will make it as small as iPods.• AR is a new and complex medium to discover!

01/07/2010 15:36 14

The Vision of Yolande Kolstee

Visualization in Art and Design

01/07/2010 15:36 15

Royal Academy of Art (KABK)

The Hague

The Netherlands

www.ARLAB.nl

Undergraduate / Bachelors

� Design: Textile, Fashion, Interior,

Furniture, Graphic, Interactive Media

� Fine Arts & ArtScience

� Photography

Graduate / Masters

� Type and Media

� ArtScience

� Interior Architecture

� Research in the Arts

� Industrial Design

01/07/2010 15:36 16

Delft University of Technology

Delft

The Netherlands

Undergraduate, Graduate & Doctorate

Faculties:

� Aerospace Engineering

� Applied Sciences

� Architecture

� Civil Engineering and Geosciences

� Electrical Engineering, Mathematics & Computer Sc.

� Industrial Design Engineering

� Mechanical, Maritime & Materials Engineering

� Technology, Policy and Management

01/07/2010 15:36 17

The fusion of perception of real and virtual imagesThe AR Lab is for all students, also outside students, who need help in the AR domain.

Institutes of cultural heritage come to us, like Museums:•Van Gogh•Boijmans van Beuningen•Stedelijk Museum•Kröller-Müller•Escher

Manifestations of:Multi-mediaModern-ArtICT

01/07/2010 15:36 18

For example we used AR to show animated chairs and tables in the 2008 exhibition Salone del Mobile in Milan, Italy

01/07/2010 15:36 19

AR was used to demonstrate the construction process

01/07/2010 15:36 20

AR can be used to demonstrate the possibilities of the product

Melanie Luchtenveld

01/07/2010 15:36 21

The virtual objects can be studied from all viewpoints

01/07/2010 15:36 22

Transportation and duplication of furniture is quite easy

01/07/2010 15:36 23

Connecting with the virtual world, experiments with interfaces

Audience likes AR & Interaction �Wii remote controller�RFID�Data glove: 5 flexor strips in fingers�Markers 'stand-alone' or incorporated�Fragments of pottery

01/07/2010 15:36 24

We use RFID technology to let (tagged) physical objects influence virtual objects

01/07/2010 15:36 25

We use a data-glove to manipulate virtual manipulated objects. Here to realize a virtual hand-puppet

01/07/2010 15:36 26

One can interact with virtual objects in several ways

One can us real physical objects to change behavior of virtual puppets e.g.

Bath-sponge � towel and foam

Pot-holder � cook

Fay-wand � blow fish is crowned

Cigarettes � blow fish blows

01/07/2010 15:36 27

Interaction can also be performed by moving the markers

01/07/2010 15:36 28

Interaction with a WII mote

� Escher in the Palace:� Wii remote controller:

throw a visual to thewall

01/07/2010 15:36 29

Headset Design

Students of the Royal Academy designed a new AR headset based on the VR/AR headset of the Dutch SME Cybermind who made a working version of the student’s mock-up

01/07/2010 15:36 30

Design requirements:� More ease to wear� Don't isolate the user� Don't block the reality� Contact with the environment� Unique experience / attractive

01/07/2010 15:36 31

Headset 'George'

� camera at the right side� inertia tracker left� open view below� wide screen� Side view

01/07/2010 15:36 32

Various other designs have been made, but not yet prototyped

Design Niels Mulder

01/07/2010 15:36 33

What to display?

Man made:There is a variety of 3D design programs and skilled and talented people who master this:Fine Art Artists, Animators, 3D Designers, Architects

Computer Generated:� 3D imagery (industrial) machines

especially from the bio-medical research� CT scan generated 3D images, yield an

inexhaustible amount of CT images

Data visualization:� There are various huge databases, e.g. geo

information, annual reports

01/07/2010 15:36 34

Art: Drops of White

Marina de Haas

01/07/2010 15:36 35

Geo Info

Data Visualization of Acid level in the first AR MAP on behalf of GSDI 11 World Conference, RotterdamJune 16-20 2009

Global Spatial Data Infrastructure association

01/07/2010 15:36 36

AR Art: first experiment for general public

Marina de Haas

01/07/2010 15:36 37

Cultural Heritage: Displaying computer generated data:Using bio-medical imaging equipment: CT scans for visualizing sgraffito-pottery:

Pre industrial original (1250-1450)Post industrial means Scanning- Digitalizing

3D printing � physical copy3D Archiving � virtual copy

01/07/2010 15:36 38

01/07/2010 15:36 39

Lessons learned

� Let the public view on a beamer what is seen by the HMD

� The “empty” third dimension is very useful for information display and interaction and detaches the application from gravity. One can have art or any info in the air ( e.g. drops of white)

� Motion of the virtual objects covers up for misalignment and jitter

� Design packages such as Cinema 4D make design with animated figures possible. For real 3D animated films with large plots, we are busy to incorporate game engines like Unity

01/07/2010 15:36 40

� Headset based AR is at its best when a full immersive experience is required and people can walk around or within larger objects; buildings, DNA, airplanes to be refurbished

� For outdoor AR it is necessary that the ambient light intensity and the intensity of the LCD displays on the headset is in balance (automatic sunglasses)

� Augmented Reality can be fruitfully used to attract a broad public to displays of cultural heritage. Its narrative power is huge.

� The collaboration between researcher in the area of image processing with artists, designers and curators appeared to be very fruitful and has lead to many amazing productions and exhibitions

01/07/2010 15:36 41

“It begins with a blessingand it ends with a curse;making life easy,by making it worse.”

‘Why more realities?’

the old man in Milan said:

‘Live is already so complicated’.