When the movie Night At The Museum 2 launched in May last year, Sydney’s Sunday Telegraph did something no other paper in

the world had done before.Software called ‘Papermotion’ allowed

readers to hold up their newspaper to a webcam that seemingly brought the film’s characters instantly to life in their hands. It was a startling marketing ploy, and one the paper’s owner, News Corp, says it’s eager to repeat — perhaps with news stories, competitions and games. And they’re hardly alone. Papermotion is just one variation of Augmented Reality — or AR as it’s popularly known. It’s a technology that’s been in the wings for decades, but whose ‘Kodak moment’ has definitely arrived.

AR works by adding layers of virtual information and graphics on to the real world. It usually operates over video or digital images, but physical objects can be enhanced as well. Because of the processing power involved, early examples were used

mainly by television production companies — think of those virtual lanes overlaid on Olympic swimming events. But today we all have the required processing power in our pockets thanks to the spread of smart phones with video, GPS and accelerometers, while our powerful PCs and laptops at home, together with fairly open development software, means AR is now within reach of everyone.

And it’s expected to get huge — Juniper Research in the States says it’ll generate $700 million in revenue by 2014, as big business embeds AR into ads, books, magazines, phones, CD covers, games… there seems no end to its possibilities.

The waning popularity of virtual reality — a once white-hot trend that failed to hold our undivided attention once social networking kicked in — is also helping. While the biggest virtual world, Second Life, says it’s still growing, many of the others (and there were 200 of them just for kids at one point) have faded in to cyberspace.

“Virtual reality locks you into the artificial, while AR is a real-world experience — and that’s where we spend most of our lives,” says Bruce Thomas from The Wearable Computer Lab at Adelaide University. “Like the internet, though, no one can really predict where AR is going, but its everyday accessibility means it’s got a lot of potential. Obviously it will get a lot of trivial uses too, but as connectivity becomes ubiquitous it also looks like it’s going to be incredibly useful.”

AR’s in handIt’s something the phone and technology companies were quick to understand. AR-based apps are now the rage with Apple, Nokia and Google’s Android all bringing out AR-related applications at a cracking pace, with names like Wikitude, SREngine, Worksnug, Pocket Universe, SkyMap and Junaio, to name only a few. All of these share the ability to blend a phone’s camera with various bits of online data, from subway

Augmented Reality is mashing up the virtual and the real in weird and exciting ways. Steve Freeth finds out why our perspective is changing.

TOP — Rocket Racing League’s X Racers: real planes, AR-helmeted pilots competing across skies filled with virtual tracks; spectators can take virtual part using iPhone Apps held to the sky;ABOVE LEFT — Night At The Museum: bad movie, cool AR from the DVD jacket;ABOVE RIGHT — Lego increased sales using AR on-screen ‘instant construction’ at sales centres

systems to maps, in real time. The iPhone’s Yelp, for instance, opens up Monocle with a few shakes of the handset to download reviews of nearby restaurants and bars on the street you’re walking down, while TwittARound can show you who’s twittering and from where.

One of the fastest growing, though, is Layar, the so-called ‘reality browser’ (see p44) that superimposes details from a growing army of sites and apps, including Wikipedia, Flickr, Google and YouTube. You can even sort the layers by country, with Australia’s store already holding radar maps from NAB, Red Rooster, the Powerhouse Museum and AusWiki.

But for a glimpse of the future, go no further than Swedish technology designers The Amazing Tribe (TAT for short), which has caused something of a web sensation with its Recognizr prototype. Already being ported to Android, though with no direct sales to consumers planned, Recognizr combines facial recognition software and AR to identify people and then pull in personal data from Facebook, MySpace or anywhere else it can find your info lurking online.

No arguments, then, with a recent media statement from a company spokesman who said “this takes social networking to a whole new level”.

Show me the moneyNews Corp may have been first in the newspaper AR game, but marketers and advertisers around the globe were already ahead of them.

A whole swathe of big, international companies have tied their products to the high-tech phenomenon in the last year or so, including Benetton, Calvin Klein, General

BELOW — Sony’s EyePet uses the PlayStation 3’s EyeCam to trigger dozens of clever AR inter-actions. Expect a second dose of EyePet to arrive once the Ps3’s ‘Move’ motion controller launches

“ObviOuSly AuGMENTED REAliTy will GET A lOT OF TRiviAl uSES... buT AS cONNEcTiviTy bEcOMES ubiquiTOuS iT’S GOiNG TO bE iNcREDibly uSEFul...”

Augmented reAlity

40 GEARE #62 www.avhub.com.au 41

Augmented reAlity

in a tough market. Esquire, Wallpaper, Grazia and GQ magazines have all brought out recent ‘AR’ editions overseas, with Esquire’s delivering a walking, talking Robert Downey Junior on the cover, as well as other ‘live’ segments inside (once you had downloaded the free software).

Window shoppingNo wonder retailers can see dollar signs and are lining up to incorporate the AR experience both inside and outside shops, or from the other side of the PC.

One company that has been fast off the mark is Lego, the international toy company, which has been installing augmented reality kiosks throughout their stores and sales points worldwide. Shoppers simply take the toy box they’re interested in to the kiosk screen to watch it being put together in 3D. It’s an experience the company says is stimulating shoppers to buy.

There’s also a push to make mirrors in stores AR-enabled too. Shiseido, the make-up giant, has been rolling out a Digital Cosmetic Mirror in Tokyo malls that can match products to skin type, apply them with AR tools and then print out head shots with the corresponding shopping list — all Shiseido naturally.

Plenty of retailers are making use of their windows to grab attention in the same way. In London both Hugo Boss, the global fashion house, and upmarket jewellers Tissot have recently created interactive windows to lure in pedestrians. Tissot’s gamble involves handing out paper watches to people as they pass by so that they can then see how the company’s ‘real’ watches look on them.

And in a sign of just how important AR might be as a retail ‘touchpoint’, ShopSavvy from the company Big in Japan lets you scan a bar code with your phone for product reviews, price comparisons and availability, online ordering or to notify a friend. It’s working too — at the end of last year more than three million people were actively using the technology, 91% of them while in a store. In the US there’s an iPhone App called GoodGuide which uses your phone-cam’s image of a barcode to deliver details of the green credentials for some 65,000 high-street products. But why bother schlepping to the store in the first place? Companies like Zugara, Glasses Direct and

earthmine mines marsEarthmine calls its technology “reality indexing” — the overlaying of AR data onto detailed 3D panoramic images. While Earthmine has to collect the local terrain details the old-fashioned way (by taking its automated vehicle-based camera arrays down every road, alley and laneway in the area to be mapped), it does so in spectacular quality beyond anything currently possible using Google-based mapping. And that’s thanks to the 3D data-generation software and algorithms it has licensed from NASA’s Jet Propulsion Laboratory.

Each and every mapped pixel contains real-world latitude, longitude, and elevation information, so that everything within an image can be easily

located, measured and modelled using simple points, lines, or polygons. No detail is left undocumented, allowing visualisation of everything from overhead power lines and multistorey buildings, to underlying road and curb features.

The technology was originally developed for the wide-angle stereo cameras mounted on NASA’s twin Mars Exploration Rovers (MERs) launched in 2003. The camera technology creates a 3D representation of the local terrain to allow autonomous routing of the MERs through the Martian environment, and it obviously works — the two robot geologists are still out there, having far outlasted their intended 90 days of operation. (‘Opportunity’ is still chugging its way across the planet and returning data to Earth, while ‘Spirit’ went silent in March 2010, hopefully recharging its batteries ready to reawaken at some future date.)

Earthmine’s earthbound use of the technology has extended to the creation of virtual billboards visible only through smartphones, as well as solutions for real estate, engineers and planners, architects and public transport planning. Images: NASA/JPL-Caltech/Earthmine

Electric, Hallmark, CNN, Nissan and Nokia… and the list goes on.

Typical of their approach was the tie-in between Coca Cola and the movie behemoth Avatar — no stranger to AR itself of course. Special cans and bottles of Coke Zero activated a PC’s webcam to link to an on-screen 3D image of one of the film’s helicopters, which could then be controlled by moving the can or bottle around. But such icon-driven AR is already considered

feebly out of date by most AR developers. You can get ready for a lot more in-your-face campaigns. We may still be some distance from the film Minority Report where electronic billboards and avatars call out with tailored come-ons, but it’s surely coming. Geo-tagging and other forms of geospatial markers are already in use, and are expected to provide the basis for more interactive AR, especially to our phones. Media companies are also hoping AR can help them stand out

ABOVE —1: The Phillies’ Ryan Howard on a Topps AR baseball card 2: In retailing, face and motion detection are being combined with AR overlays to virtually dress or make-up customers

3: In the automotive sector, companies including BMW are developing informational AR overlays for the windscreen, and also complete AR servicing systems where mechanics are guided through engine repairs using AR headsets

4: In industrial manufacturing, AR can works for robots as well as headset-wearing workers to provide clear instructions

5: Making AR — Metaio’s Unifeye Design 2.0 enables design of augmented reality presentations and marketing applications

Ray Ban already let you try on clothes or sunglasses straight to your own face or body via websites using AR and motion sensing.

Press PlayNone of the AR possibilities have been lost on the people wanting us to have fun. The gaming industry would love to take the 3D screen experience out into the real

world. It’s happening of course — with much R&D under way around the world — but the continuing reliance on the webcam, markers and handheld devices in games like Invizimals, Pacman, Streekstation, memoARy or ARhrrrrZombies suggest it’s still got some way to go.

The holy grail for the industry will be sleek, wearable computers (like AR goggles

and headsets), but even without these, games like Parrot’s AR.Drone — shown at CES and slated to be released some time this year (see p45) — shows what the potential can be with the right hardware in combination with smart phones. And there’s still a lot out there to keep us — or at least the kids — entertained. The company Topps in the States, for instance, has come out with

1

3 4

2

5

Augmented reAlity

42 GEARE #62 www.avhub.com.au 43

Augmented reAlity

3D baseball trading cards, and we looked at camera-initiated AR on packs of Top Trumps recently in Geare. Adidas has embedded AR into a line of sneakers that sparks an online 3D game, and of course Sony had something of a hit with EyePet, a cute AR construct run through the PlayStation 3 that is able to trigger all sorts of 3D add-ons, though for our money Sony spoiled things by locking up the best tricks until you’ve navigated all the sometimes tedious gameplay. Expect Eyepet to return once the PS3’s ‘Move’ motion controller hits the market.

Book and music publishers are also beginning to explore how their products can gain a little wow with AR. In Germany ArsEdition has an upcoming interactive 3D book called Aliens & UFOs, while WissenMedia is using AR with the Atlantica series to bring said planet to life.

But it might well be in the real world of sports that AR gets to really shine. The futuristic-sounding Rocket Racing League certainly hopes so. From 2012 (they hope Australia will be one of the first countries to take part) the group plans to have real pilots wearing AR-enabled helmets in planes called

X Racers competing across skies filled with virtual tracks. Spectators will watch the action on giant screens with virtual overlays of data, or even take virtual part using Apps available on the iPhone held up to the sky.

I can see clearly nowSo far, so terribly trivial you might be thinking. But AR does have a serious side.

In fact AR may hit its deepest pay dirt in industrial and service applications as companies adapt the concepts to help us do real, everyday or complex tasks much more safely and efficiently. It’s still early days for

much of this, but there are some exciting ideas out there.

Car companies like GM, BMW and Mercedes have been flirting with AR- infused windscreens for some time, so that information like distance, weather or traffic conditions appear in front of drivers without the distraction of using phones or onboard computers.

They’re also hoping AR could be some help under the hood. Metaio, an AR-centric company, has already developed the Unifeye Design software that can allow 3D modelling of cars and engines, but which they hope will have much broader industrial applications.

It’s an approach that Columbia University has pursued with their Augmented Reality for Maintenance & Repair, a project that hopes to harness wearable hardware and AR for engine visualisation, so that repairs can be done in situ using online expertise and back-up. A BMW research project shows a service engineer donning glasses and being led through procedures using AR overlays on the engine — “Remove two screws here…” etc.

The health, education and military sectors are taking interest as well. Siemens, the electronics giant, says it’s exploring how AR could be applied directly in areas like health, while Canadian company Arcane Technologies is taking that route as well, using projected 3D data for behavioural and surgical treatments, as well as training. In fact the company has developed the Mirage System, a wearable device that lets you apply AR to tailored situations; they are putting it to work to help people overcome phobias or in factories for Guided Assembly.

Too much information?Not everyone is convinced that AR will be much more than a passing fad once the initial eye-popping thrill has gone.

Some of the physical constraints seem a drag for some. For a start it’s still a technology very much attached to handsets and web cams, and that can be both clunky and inconvenient, while GPS also has limits, though much more precise positioning is at hand once our skies fill up with navigation satellites.

WHERE AM I? — Applications such as Layar (left), Wikitude (above) and Worksnug (right) use a mobile phone’s camera, compass and GPS data to identify location and field of view, then retrieve data based on those coordinates

LEFT — Some forms of AR appear fairly conventional, such as Yelp’s system of alerting you to restaurant and bar recommendations relevant to your location. While valuable if recommendations are made by early adopting technology enthusiasts, it’s not hard to see how quickly such as system could be overloaded with advertising as well as genuine recommendations

Some critics also suggest that we may tire of having too much information in our lives, strapping us into devices and cyberspace at the expense of real-world experiences — think Terminator in the milk bar. Others suggest that apps like Recognizr will open us ever more to identity theft and online fraud.

But many of the same arguments were levelled at the internet. More likely AR will become one element of new applications that seamlessly integrate data with the real world. That’s likely to be more probable through the use of projected AR, or wearable devices like goggles — a process being called Mediated Reality. You can see the seeds of that already

ParrOt taKes ar intO flightToy helicopters and planes are nothing new — but toys with Wi-Fi control, onboard camera and augmented reality definitely are. The AR.Drone ‘quadracopter’ uses two cameras. The first is under the main structure, connected to an Inertial Measurement Unit which measures speed and allows it to perform stationary flight and automatically compensate for turbulences due to the wind during outside flights. The second camera is at the front of the quadricopter, broadcasting and streaming to an iPhone or iPad, so you see what a pilot would see. You can also fit the AR.Drone with gyroscopic stabilisers.

So far, pretty thrilling. But there’s more. You can battle various electronic enemies overlaid onto your live camera feed. Place a marker on a chair and the AR.Drone’s will recognise it and replace it with a fierce robot. Get two AR.Drones and they will recognise each other, so you can blow three shades of hell out of each other and recreate the flying battle scenes of Avatar.

All this takes the AR.Done’s creator, the French company Parrot, into rather new territory. Their expertise is in Bluetooth and Wi-Fi, applied to hands-free car units and music-streaming speakers, but the AR.Drone’s abilities are, of course, all communications based. With hopes of release in time for Christmas, expect tales of privacy invasion as teenagers use the AR.Drone for hovering outside windows, over the local sunbathing spots etc… More: http://ardrone.parrot.com

“GEO-TAGGiNG AND other forms of geospatial markers are already in use, and are expected to provide the basis for more interactive ar...”

beginning to take shape. Another Canadian company, Eyetap, has a sleek if still not commercially-manufactured AR-based eyepiece it has been pushing for some time, while over at the Massachusetts University of Technology they’ve developed a prototype called Sixth Sense that is light and wearable — if a little under-developed — and projects AR data on to the real world.

The virtual seepage into the real world is here, and about to get faster, leading some to call this the beginning of the ‘outernet’. Whatever it’s called, let’s just hope the AR spectacles look a lot better than the 3D ones. Steve Freeth

Augmented reAlity

44 GEARE #62 www.avhub.com.au 45

Augmented reAlity