iPhone Augmented Reality

Jonathan Saggaujonathan@jonathansaggau.com

@jonmarimba

Jonathan Blocksomblocksom@gmail.com

@jblocksom

About Us

• Jonathan Saggau (Also No PhD)

• Jonathan Blocksom(No PhD)

VR / AR History

• 1965Sutherland

VR / AR History

• 1995 - UNC,Nintendo

• who is that? ------>

Stella!

Existing AR Apps:Theolodite

• Sort of like fancy binoculars

DishPointer

Existing AR Apps:Layar

• Overlay markers on the world

Augmented Surreality:Numen

AR Drone

• iPhone controlled remote flyer

• Sends video stream to phone over Wifi

• http://ardrone.parrot.com/parrot-ar-drone/

More Existing AR Apps

• http://mashable.com/2009/12/05/augmented-reality-iphone/

• http://www.youtube.com/watch?v=ps49T0iJwVg (acrossair)

• Augmentia

• Simple Geo

• smaato

• Layar

• Wikitude

From zero to hero: starting with the “View based application” template and ending with a tagged landmark

• Hello (Augmented) World

• Picture Level (rotation about 1 axis)

• Artificial Horizon (rotation about 2 axes)

• Artificial Horizon + Compass (add cardinal direction / translation)

• GPS Locator: tagging a landmark (add location / translation)

Baby steps to a better reality : Hello (Augmented) World

• Goal: Overlay some text onto the live camera image

• UIImagePicker + overlay

• Step 1: collect underwear

r3

UIImagePicker overlay-(void)viewDidLoad{ [super viewDidLoad]; picker = [[UIImagePickerController alloc] init]; picker.sourceType = UIImagePickerControllerSourceTypeCamera; picker.showsCameraControls = NO; picker.navigationBarHidden = YES; picker.wantsFullScreenLayout = YES; //1:1.124.. = aspect ratio of the screen picker.cameraViewTransform = CGAffineTransformScale(picker.cameraViewTransform, 1.0f, 1.12412f);}

-(void)viewDidAppear:(BOOL)animated{ [super viewDidAppear:animated]; [self presentModalViewController:picker animated:NO]; UILabel *aLabel = [[UILabel alloc] initWithFrame:CGRectMake(10., 50., 300., 30.)]; [aLabel setText:@"Hello augmented world!"]; [aLabel setBackgroundColor:[UIColor clearColor]]; [aLabel setTextColor:[UIColor whiteColor]]; [aLabel setFont:[UIFont boldSystemFontOfSize:12]]; picker.cameraOverlayView = aLabel;}

Picture (er, shopping cart) Level

• Goal: Overlay a level bar on the center of the live camera view

• Requires: real time info regarding the rotation of the camera about the Z-axis

• Adds UIAccelerometer

r7

-(void)viewDidAppear:(BOOL)animated{ [super viewDidAppear:animated]; [self presentModalViewController:picker animated:NO]; label = [[UILabel alloc] initWithFrame:CGRectMake(10., 50., 300., 30.)]; [label setText:@"Hello augmented world!"]; [label setBackgroundColor:[UIColor clearColor]]; [label setTextColor:[UIColor whiteColor]]; [label setFont:[UIFont boldSystemFontOfSize:12]]; CGRect viewFrame = self.view.frame; viewFrame.origin.x = viewFrame.origin.y = 0.0; UIView *overlayView = [[UIView alloc] initWithFrame:viewFrame]; [overlayView addSubview:label]; [overlayView bringSubviewToFront:label]; // Load the image to show in the overlay: UIImage *overlayGraphic = [UIImage imageNamed:@"artificialHorizon.png"]; overlayGraphicView = [[UIImageView alloc] initWithImage:overlayGraphic]; overlayGraphicView.frame = CGRectMake(30, 100, 260, 200); [overlayView addSubview:overlayGraphicView]; picker.cameraOverlayView = overlayView; [overlayView release]; [[UIAccelerometer sharedAccelerometer] setDelegate:self];}

#pragma mark UIAccelerometerDelegate- (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration;{ double angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; double angleInDeg = angle * 180.0f/M_PI; [label setText: [NSString stringWithFormat:@"Angle: %f", angleInDeg]]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle);}

Video will go here

Artificial Horizon

• Goal: Keep the bar on the horizon (not just level)

• Compensate for rotation along x-axis

• Need to know field of view of camera to find horizon position

• ~53 degrees vertical

• ~37.5 degrees horizontal

r8

iPhone View Angles53 Degrees Vertical*37.5 Degrees Horizontal

*Close to this, but not exactly on the iPhone 4

#pragma mark UIAccelerometerDelegate- (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration;{ overlayGraphicView.transform = CGAffineTransformIdentity; double vertAngle = -atan2(acceleration.y, acceleration.z) - M_PI/2.0; double vertAngleInDeg = vertAngle * 180.0f/M_PI;

[label setText: [NSString stringWithFormat:@"V-Angle: %f", vertAngleInDeg]];

CGRect overlayFrame = [overlayGraphicView frame]; overlayFrame.origin.y = 240.0 - 537.8 * sin(vertAngle); [overlayGraphicView setFrame:overlayFrame]; double angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; //double angleInDeg = angle * 180.0f/M_PI; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); }

Directional Horizon Marker

• Goal: Artificial horizon pegged to a specific cardinal direction (North)

• Adds compass direction using CLLocationManager

r10

locationManager = [[CLLocationManager alloc] init]; [locationManager setDelegate:self]; [locationManager startUpdatingHeading];

- (void)updateUI{ [label setText: [NSString stringWithFormat:@"H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f", angleInDeg, vertAngleInDeg, magCompassHeadingInDeg]]; CGPoint overlayCenter = [overlayGraphicView center]; overlayCenter.y = 240.0 - 537.8 * sin(vertAngle); overlayCenter.x = 160.0 - 497.8 * sin((magCompassHeadingInDeg) * (M_PI / 180.0)); [overlayGraphicView setCenter:overlayCenter]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle);}

#pragma mark UIAccelerometerDelegate- (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration;{ vertAngle = -atan2(acceleration.y, acceleration.z) - M_PI/2.0; vertAngleInDeg = vertAngle * 180.0f/M_PI; angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; angleInDeg = angle * 180.0f/M_PI; [self updateUI];}

#pragma mark CLLocationManagerDelegate- (void)locationManager:(CLLocationManager *)manager didUpdateHeading:(CLHeading *)newHeading;{ magCompassHeadingInDeg = [newHeading magneticHeading]; [self updateUI];}

GPS Hotel Locator

• Goal: Speakers must find Hotel in any (um)state, so we’ll tag it for them.

• Add GPS location with CLLocationManager

• Compensate for “the hotel is over there and I’m looking over here”

In other words:

Given the position of the hotel, the position of the iPhone, and the direction the iPhone is pointing,

we want to know the cardinal direction of thehotel relative to the iPhone’s pointing direction

and the elevation of the hotel relative the iPhone’spointing direction in order to overlay a label

onto the hotel in real time.

- (void)updateUI{ CLLocation *marriot = [[LocationTools class] locationForMarriott]; float directionToMarriott = [[LocationTools class] angleFromCoordinate:[self.currentLocation coordinate] toCoordinate:[marriot coordinate]]; CLLocationDistance dist = [self.currentLocation getDistanceFrom:marriot]; double deltaAlt = [marriot altitude] - [self.currentLocation altitude]; //verticalAccuracy of -1 == invalid if ([self.currentLocation verticalAccuracy] < 0) deltaAlt = 0; double vertAngleToMarriott = atan2(deltaAlt, dist); [marriotLabel setText:[NSString stringWithFormat:@"DirHotel:%5.1f Dist: %5f VertAngle: %f", directionToMarriott * (180.0 / M_PI), dist, vertAngleToMarriott * (180.0 / M_PI)]]; [anglesLabel setText: [NSString stringWithFormat:@"Horizon H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f", angleInDeg, vertAngleInDeg, magCompassHeadingInDeg]]; double relativeDirectionToMarriott = magCompassHeadingInDeg * (M_PI / 180.) - directionToMarriott; if (relativeDirectionToMarriott < (-M_PI / 2.0)) relativeDirectionToMarriott = (-M_PI / 2.0); if (relativeDirectionToMarriott > (M_PI / 2.0)) relativeDirectionToMarriott = (M_PI / 2.0); double relativeVertAngleToMarriott = vertAngleToMarriott - vertAngle; CGPoint overlayCenter = [overlayGraphicView center]; overlayCenter.y = 240.0 - 537.8 * sin(relativeVertAngleToMarriott); overlayCenter.x = 160.0 - 497.8 * sin(relativeDirectionToMarriott); [overlayGraphicView setCenter:overlayCenter]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle);}

Gotchas

• GPS and compass haven’t enough accuracy to place advertisements or labels on near objects.

• On a “bad day” your iPhone’s version of “here” might be a 1/2 kilometer circle around a cell tower... or worse.

• Location updates will drain battery... Quickly.

• But the dream is alive.

“Best practices... ” or “stuff we noticed making a demo AR app”

• Don’t try to tag a near object, inaccuracies compound as you near the target.

• Check altitude / location accuracy: Phone can’t get altitude w/o good GPS fix. See documentation for accuracy (check for -1)

• The broad side of the barn is easier to hit.

• Bottom line: “here” isn’t here, yet.

Download the code

• http://code.google.com/p/ar360/

iPhone OS 3 Stack

!"#$%&'(")*%+,%-./%0#12*)3*4%5*67&38%099%

:;;261*'&<=*$>")3$"77*$%?@&4*"%&)9#3A%

B"<6C")D6)61*$%?E',F%>"296GGA%

:;0<<*7*$"2*3*$%4*7*163*%?0<<*7*$"2*3*$A%

iOS 4 Ideal

!"#$%&'(")*%+,%-./%0#12*)3*4%5*67&38%099%

0:;"#)46<")%=>&4*"%&)9#3?%

@"A6<")B6)61*$%=C',D%E"296FF?%

E"$*%B"<")%=0AA*7*$"2*3*$D%C8$"F?%

AV Foundation

• Allows processing of the frame before display

• Very complicated to setup

• Check out the WWDC samples and video

What’s next?

• Smoothing / Filtering Sensor Data (wobble wobble)

• Full-on 3D AR world -- Dancing hippos in swimming pools with Dolphins...

• Image tracking to mitigate the swings

• ?

• Profit!

How we might transition to a Full virtual 3D World

• Need useful local coordinate system to map onto OpenGL; we’re pretty close with the demo code, actually.

• Match OpenGL transform matrices with current position relative to some scene we might want to add to the real world, camera parameters; GLULookAt...

• Need to not mind that compositing could be very slow...

ARKit :Existing AR Toolkit for iPhone

• http://github.com/zac/iphonearkit/

• Marker based

• Simple CoreLocation - based data model

• Hasn’t been updated for a while

What counts as AR?

• Video with... ?

• Overlay?

• Must be Interactive?

• GPS?

• Modify the virtual environment?

AR News Sources

• Bruce Sterling’s Beyond the Beyond bloghttp://www.wired.com/beyond_the_beyond/

• Tish Shute:http://www.ugotrade.com/

• Twitter streams:@anselm, @bruces, @tishshute, @brady

• O’Reilly Radar:http://radar.oreilly.com