How Eye Tracking Will TOTALLY Change the Way You Game

11/7/2015 How Eye Tracking Will TOTALLY Change the Way You Game. — The Future of Gaming — Medium

https://medium.com/the-future-of-gaming/how-eye-tracking-will-totally-change-the-way-you-game-193126bbbba4 1/25

How Eye Tracking Will TOTALLY Change theWay You Game.By Russ Pitts (Flying Sauce Media, Polygon.com, The Escapist)

hey’re just kids, really. Standing around the table, armspoking in and out of my field of view as they excitedly

babble in French-scented English, their youthful enthusiasmcan’t help but be infectious. This one wants me to trysomething new in their game, that one wants me to knowthe game isn’t finished. Their voices rise and fall as eachtakes a turn. But none of it distracts me from what’shappening on the screen.

T

https://medium.com/u/1667ac75b34e



I’m playing a very early build of a video game in which theplayer navigates through a maze of caverns, encounteringmonsters and spirits, and trying to solve a (hopefully)spooky mystery. It will be a horror game, like many others.But unlike any other, you will play it with your eyes.

As I turn a corner in the maze, my eyes flick up and left andthe view of the screen shifts with them. It’s disorienting atfirst, but within seconds I’m used to it. I’m holding a videogame controller in my hands. My left thumb is controllingthe direction I am moving. My right thumb is controlling aflashlight, which allows me to see better in the darkenedcave. And my eyes … my eyes are controlling where thegame’s “camera” points — what would normally becontrolled with the right thumbstick.

And my eyes … my eyes are controlling where thegame’s “camera” points — what would normally becontrolled with the right thumbstick.

The kids are all right — their youthful enthusiasm in the corner of Tobii’s HQ is an understatement.



It is therefore possible for the light to shine in a completelydifferent direction from where I am looking, just as wouldhappen in real life. And the effect is startling and new.

I turn a corner and see something out of the corner of myeye. I glance at it, the game refocuses on that point, but myhand moves more slowly and I am looking at something toodark to see until the thumb controlling the flashlight catchesup. And in that moment of disoriented darkness … anythingcan happen. Boo!

Is it a ghost I see before me, or merely shadow? That’s whatwill make for a (hopefully) scary, fun horror game.(Nowavailable for download) And what makes its unique take onthe horror genre possible is an eye tracker, by Tobii Tech.

An Overnight Success, 15 Years LaterTobii may be one of the most successful technologycompanies you’ve never heard of. If I’m being honest, Ihadn’t heard of them before I spoke with them about thisstory either.

Arriving at Stockholm’s Arlanda airport, I explained tocustoms officials I was in Sweden on business. They askedfor whom I was working. I said “Tobii. It’s a technologycompany — “

“We know,” they interrupted.

OK. They know. But that’s in Sweden, where Tobii employsover 300 people and has attracted upwards of $50 million ininvestment. In the rest of the world? You would probablyhave never heard of them. Unless you’re one of hundreds ofuniversity researchers, or thousands of people with

http://www.tobii.com/xperience/apps/glimpse-of-fear/?utm_source=medium.com&utm_medium=blog&utm_campaign=russmedium



disabilities who have already been using Tobii’s line of eyetrackers.

This company coming seemingly out of nowhere with astand-alone eye-tracking device in a partnership with SteelSeries, an eye-tracker-enabled gaming laptop in partnershipwith MSI, and a growing number software partnerships withcompanies like Ubisoft and Avalanche might be a new forcein gaming, but it’s been studying eyeballs — and trackingthem — for a long, long time. When you play Assassin’sCreed: Rogue, The Hunter, The Hunter: Primal and manymore games to come (they showed me some, but I can’t talkabout them), you will experience what data researchers andability engineers have known for over a decade: Your eyesknow what you’re thinking before your hands do.

Tobii’s HQ in Stockholm, Sweden

http://www.tobii.com/xperience/apps/assassins-creed-rogue/?utm_source=newsletter&utm_medium=medium.com&utm_campaign=russmedium

http://tobii.com/hunter?utm_source=medium&utm_medium=blog&utm_campaign=russmedia



You will experience what data researchers and abilityengineers have known for over a decade: Your eyesknow what you’re thinking before your hands do.

My time at Tobii is full of interesting experiences, but onething sticks out — and it happens multiple times — thepeople here are almost supernaturally aware of what I’mthinking. In multiple interviews, Tobii employees willcomment on how I am interacting with them, how Imaintain eye contact while they are speaking, to encouragethem to continue; how I nod at them to indicate that I amlistening; and smile to influence them to expand on whatthey might be saying. Subtle interview tricks I learned solong ago I’m no longer aware of doing them. But the peopleI talk to at Tobii are aware. They’re aware and responding.Because they’re watching my eyes. And they’re programmingcomputers to do the exact same thing.

Tobii was founded in 2001 by a Henrik Eskilsson, JohnElvesjo and Marten Skogö. John was working in a researchlab on technology to detect bubbles in various liquids thensomeone had the bright idea: You know what’s also like abubble? And then came the idea: Let’s build a device thatcan track the human eye, then put an eye tracker into everycomputer. Tobii was born.

Tobii’s very �st eye tracker (circa ��)



“The dream is for us make every computer know what youwant to interact with before you even touch any controller,”says Tobii Tech’s vice president of product, HenrikJohansson. “We are making steady progress, we havecomputers in our labs where you can just look at an icon andplace a finger on a touch pad to click, a similar interaction toon a phone or a tablet.”

Currently, Tobii is the market leader. In fact, no one else iseven close.

Part of that has to do with a divergence in the evolution ofcomputing technology. When Tobii was founded, no one(except maybe Steve Jobs) expected tablets and phones tobecome the most common computing devices. But they did,and so Tobii changed course.

The new goal, says Johansson: “To be in every laptop andtablet and mobile and car. That’s what we’re working on. It’sTobii Tech’s task, to make that happen.”

The Window to Your MindTobii Tech is one of three divisions at Tobii. The first focuseson professional researchers. Scientists. Marketers.

The future of Eye Tracking will be in multiple devices



“It started out as academic research and broadenedthroughout to different types of consultants or professionalsthat want to interpret where a person is looking,” saysTobii’s Johan Bouvin. “It can be research in psychology,neurology, media design, usability, web design … a lot ofmarketing.”

The idea is that if you can tell what person is looking at,then you can most likely tell a little about what they’rethinking. What you look at is a good approximation of whatyou think. In marketing especially this is crucial. When youspend thousands of dollars on advertising, you want to besure people are paying attention.

Eye trackers attached to computer screens, or in the case ofsome research applications, worn on the face, keep track ofwhere a person’s eyeballs are pointed, and then that harddata is used to decide how effective a given whatever risk atcapturing someone’s attention.

From there Tobii started working with assistive technology,spawning a new division, Tobii Dynavox. Eye-trackingspeech computers, communication software, computeraccess tools — Tobii Dynavox makes almost everything youcould imagine that might help someone with disabilities usetechnology. And even some devices that go beyond, toimproving quality of life.



Bouvin shows me a large tablet that can be mounted onto,say a wheelchair. It has inputs for a variety of sensors thatcan be attached anywhere on the body. Even if someone isparalyzed from the neck down TobiiDyanvox device cantrack the user’s eye movements with the integrated eyetracking, and just like that someone who may not even haveuse of a hand can use a computer. And if you can imagine allof what might be done with a computer, suddenly you’veallowed for an almost limitless variety of interactions.

“Being able to use your eyes to browse the web, sendemails, create artwork, music, call friends and familyand sometimes even get back to work opens up somany doors that previously was closed for people withdisabilities.”

The main word they use is that they get some of theirindependence back.

Johan Bouvin, Product Manager at Tobii



And then there’s Tobii Tech. Tobii’s third division wasfounded to do one thing, and one thing only (and has beengranted leeway by the corporate headquarters to take asmuch as four or five years to do it): own the consumermarket for eye-tracking. This literally translates into creatingthat market. Which is what Tobii Tech, in its steadfast,matter-of-factly Swedish way is currently doing.

The Eye Tracker.

Tobii’s eye tracking works like this: infrared illuminators“light up” the area in front of the computer screen, making iteasier for the sensors to lock in on the “bubbles” of youreyes.

Without being able to accurately predict all of the differenttypes of environments a computer might be in, it’s hard to

The Tobii EyeX.

http://www.tobii.com/eyex?utm_source=medium&utm_medium=blog&utm_campaign=russmedium




program something to be able to “see” in every possiblescenario. Changes in lighting, temperature, humidity,distance, eyewear and a plethora of other variables meanthe eye tracker has to function in an almost infinitelyunpredictable variety of circumstances. The infraredilluminators level the playing field, and making thoseemitters smaller and powerful is one of the biggest tricks inadapting Tobii’s technology for consumer devices.

The other big trick is offloading the processing to the eyetracker itself. Tobii’s newest generation device does just that.With an onboard processor and infra-red sensors capable ofreading eye movements at 60–90 frames per second, thenewest eye tracker will be smaller, lighter, more powerfuland uses less power than any eye-tracking device ever made.And Tobii Tech thinks it’s the version that will finally unlockeye-tracking for consumer devices.

— Get the latest Tobii EyeX controller and developer kit—

“Basically it’s a sensor technology that’s pretty much whatyou and I do” says Bouvin. He’s giving me a tour of TobiiTech’s demo room, a space filled with all of the devices Tobiidesigns and manufactures (although most of themanufacturing is now done offsite).

http://www.tobii.com/xperience/developer/



Sitting on a pedestal in front of Bouvin is a laptop made bygaming computer maker MSI.

“It looks for eyes. It interprets where the eyes are and wherethey’re directed. Once you know where my eyes are andwhere they’re directed and you know about the objects thatare here, it’s not that difficult to guess what I’m looking at.”

The eye tracker knows, generally, what eyes look like. Itknows that an eye is a small, black circle surrounded by alarger circle of color. And it knows that what the smallercircle is pointing at is what you’re seeing.

Part of the Tobii demo room with tons of cool devices

http://www.tobii.com/tech/technology/what-is-eye-tracking/



“It’s not even a metaphor or analogy. That’s what thealgorithm looks for. You can see the reflection of light in myeyes as lines. When I move my eyes, those reflections will bein different places, compared to the pupil. That’s part of theinformation you get from the system.

“Then it’s pure trigonometry from that point on. If thissystem knows where it is with respect to the screen and itcan calculate where the eyes are and where they’re directed,that’s enough information to get a point in space. Thosevectors cross each other.”

Which is engineer speak for “the computer is reading yourmind”.

“It makes the device more aware of you. That experienceof having a machine that starts to understand not onlyyour commands, but also your attention and yourstate.” — Johan Bouvin

If you own an iPhone, you already own a device that’s awareof you. Someone calls you. The phone’s screen lights up. Youlook at the screen to see who’s calling. Let’s say you decideto answer. You touch the green button on the screen andraise the phone to your ear. The phone, using it’s opticalsensors, can tell that it is being held close to an ear, andshuts off the screen to save power. It knows you’re notlooking at the screen, because the screen is pressed to theside of your head, next to your ear. This phone knows youcan’t see from your ear, so it stops showing you things on thescreen. That’s responsive technology.

Tobii eye tracking will take this further.



“Our eyes give so many hints about what we’re about to do,our cognitive processes,” says Bouvin. “Capturing thatinteraction in the man-machine interface is so exciting,finding ways where you can get the logic, based on those eyemovements, to behave in a way that feels natural for theuser.

“Any motor action you take, anything you touch,anything you move, your eyes will be there before yourhand.”

Bouvin demonstrates by picking up a pencil from the table infront of us. First he looks at the pencil, then he moves hishand to pick it up. In the computing world, we’ve becomeused to this type of interaction, but everyone who iscurrently alive who knows how to use a computer has had totrain their mind to add a step between what they see andinteracting with it.

Natural Targeting in real life!



We’ve had to learn the motor control of seeing, and thenmoving a cursor with a mouse or a trackpad, and theninteracting.

Tobii eye tracking will remove that inter-evolutionary step,making it possible for us to interact with computers in thesame way we interact with the world. Look at a thing.Interact with the thing. No cursor required.

“What you get with eye tracking is you can substitute prettymuch all of that positioning, all that directional input,because your eyes are there before you touch,” says Bouvin.“You’re looking there. The movement and directional thingbecomes unnecessary. Same with a mouse. You’re alreadythere with your eyes. Moving the mouse cursor there is astep you can more or less do without. The only thing that’sleft is the actual action.”

Bouvin expresses this as a hope for the future, but here inTobii’s headquarters I’m seeing it in action, using a devicethat’s available to purchase now, playing a game that’salready in stores. It’s not the future: it’s now.

— How to develop with the Tobii EyeX —

The Four Immersions“There’s a fundamental problem with games,” Olsson tellsme. “You walk around their world like you’re a dog with acone around your head. And that’s a major problem. Nomatter how good you make the graphics you have noperipheral vision.”

The Tobii eye tracking can solve that problem. And I soonget a demonstration in another game, this one the recently

http://www.tobii.com/xperience/developer/?utm_source=medium&utm_medium=blog&utm_campaign=russmedium



released Assassin’s Creed: Rogue. The first AAA title to shipwith support for eye tracking.

“There’s a fundamental problem with games, You walkaround their world like you’re a dog with a cone aroundyour head. And that’s a major problem. No matter howgood you make the graphics you have no peripheralvision.” — Anders Olsson VP of Software at Tobii

I’m walking around a room in a laboratory, one of the placesin the fictional world of Assassin’s Creed where thememories of your long-dead assassin ancestors are beingpulled from your DNA or whatever (I don’t play a lot ofAssassin’s Creed games). I’m controlling the game as Inormally would, but something curious happens when Iglance at the edge of the screen: The Tobii eye trackerdetects my gaze and begins panning the screen to the left orright, depending on where I’m looking. It’s a feeling notunlike suddenly focusing attention on what’s in yourperipheral vision. The game world in Rogue, like in theFrenchmen’s game, is responding to what I’m looking at.




The second immersion is eye contact, and it’s a harderimmersion to implement. Tobii prides itself on providing anSDK and development libraries that make it possible toimplement most eye-tracking features in a minimal amountof time. The fact that the eye tracking practically sits on topof the exciting game engine, tapping into the HID andcamera controls that already require external connectionsfor the game engine, whatever engine a developer may beusing, possible to implement features that look and feel likemagic in practically no time at all, in game developmentterms.

All of the eye tracking features implemented inAssassin’s Creed: Rogue were implemented in “abouttwo weeks.” Something almost unheard of in gamedevelopment.

Eye contact, in Tobii terms, will mean a number of things forthe games that might implement it. The simplest will makeobjects in the game world aware they’re being looked at.

See the game through the character’s eyes in Assassin’s Creed. The gameintuitively pans to where you are looking across the screen.




Similar to how an object might change color if a mousecursor is hovering over it. You can choose to interact with agame object simply by looking at it, and then use thecontroller to do whatever the game might allow you to do.More complex (but seemingly inevitable) is a game in whichcharacters you meet in the world will hold your gaze,interacting with you as a human might: by looking you inthe eye.

Nothing so complex is happening right now in Assassin’sCreed: Rogue, but the game nevertheless manages to surpriseme. My demo soon moves into the traditional realm ofAssassin’s Creed: a historical world in which I can navigateobstacles and then kill people. In true Assassin’s Creedfashion, I press X on the controller as I am moving and amable to leap over things, climb walls and trees and so forth.

But with the eye tracker I am suddenly, and almost withoutrealizing it, moving across obstacles even before I move thecontroller. The game, supplementing the right thumbstickwith the Tobii eye tracker, is responding to my gaze faster

Engage an opponent with a lingering look or enjoy cool little exchanges like afriendly nod or wink.




than I am moving the thumbstick where I want to look. Andlike the Frenchmen’s little horror game, it begins reading mymind; anticipating my moves, perfectly.

The third immersion is described to me as “naturaltargeting”, but it’s basically shooting at two things at once.

Eye tracking can make that happen. I did it. in a game calledSon of Nor I can use mouse as normal to control thedirection of my character, and at the same time shoot/throwobjects in another direction — sideways, upwards,backwards.

In fact, it makes Son of Nor so much fun I had a hard timestepping away from it, even though it’s a type of game Iwouldn’t ordinarily enjoy playing. With Tobii eye tracking,you can move around the world with the controller orkeyboard, and pick up objects and then throw them simplyby looking at them. It feels like having super powers.

Pick up a rock at the bottom right of the screen and throw it to the top left — justby looking right to left.

http://www.tobii.com/xperience/apps/son-of-nor/?utm_source=medium.com&utm_medium=blog&utm_campaign=russmedium

http://www.tobii.com/xperience/apps/son-of-nor/?utm_source=medium.com&utm_medium=blog&utm_campaign=russmedium



Imagine playing Call of Duty and shooting at whatever yourcrosshair is aimed at with your gun, but being able to throwa grenade somewhere else on the screen — simultaneously — just by looking at it.

Which makes Tobii’s fourth immersion sound like fantasy.They call it “Immersive Graphics and Sound”. And itamounts to nothing less than making a video game showyou the world in the same way you would see and hear thereal world.

Look at something in the background and the foregroundfalls out of focus, and that object becomes clearer. Look left,and the same thing happens. Your peripheral visionsuddenly takes focus, and whatever you were looking atbecome harder to see.

Graphics cards are powerful enough now, Olsson believes, tomake this happen, but no game in the world can manage itwithout eye tracking.

A simple look at a particular detail within the game will enhance its graphic orevoke a situational sound



And Tobii has announced a $1 million dollar developmentfund to help indie game developers around the worldprogram for it.

“Tobii has announced a $1 million dollar developmentfund to help indie game developers around the worldprogram for it.”

But all of the games currently available that utilize Tobii’seye tracking only capture a part of the potential of thetechnology. The first game in the world that will manage touse all of the power of the Tobii technology is a little horrorgame being made in a corner of the Tobii headquarters, by aquintet of Frenchmen.

— View even more gaming immersions —

The Window to the WorldThe Frenchmen have a workspace, deep on a bottom floor ofTobii’s spacious headquarters. It was a last-minute additionto the deal that brought them here from France. It tookthem weeks to get an apartment. They have only a fewweeks more to finish their game before heading back toschool, to graduate. This summer fellowship, as it were, anodd, little opportunity that will likely fade away into theirmemories of their time at University. But right now theycan’t believe how lucky they are.

http://www.tobii.com/boost?utm_source=medium.com&utm_medium=blog&utm_campaign=russmedium

http://developer.tobii.com/gaming-immersions/?utm_source=medium&utm_medium=blog&utm_campaign=russmedium



Interested in eye-tracking technology, the Frenchmenreached out to Tobii tech for advice, and maybe a prototype.This led to an opportunity to create a game for a showcasein Cannes. And that led to now: working at Tobii Tech, forTobii Tech, to create a game Tobii can use to promote eyetracking.

Their game, if I’m being honest, wouldn’t be noteworthywithout the eye-tracker. It’s also not finished when I get myeyes (literally) on it. But as far as horror games go, it alreadyhas a good grip on claustrophobia.

I have to imagine what it will be like when there are thingsto do and run away from, but I suspect, just from playing itin its current state, it will be intriguing. Or at least would be,just playing it with a controller.

The “dungeon” where the Frenchmen worked & played



Playing with the Tobii eye tracking, however, is anotherworldly experience. The game is responding to whereI’m looking. Tobii’s technology (after a few minutes spentcalibrating it to my eyes) is following my gaze, panningacross the screen whenever my eyes move. It’s distracting — at first — not because the outcome is undesirable, butbecause I can’t stop feeling as if the machine is reading mymind. It’s as if I’m thinking the command and the game,without needing me to move a control, is responding.Instantly.

The Frenchmen’s game (Now available for download) isdesigned to illustrate what Tobii calls “the four immersions”,or the four ways in which eye tracking can enhancegameplay.

The first is controlling the screen. It’s what allows me tomove the camera in the Frenchmen’s horror game.

Sketches for characters in Glimpse of Fear




I soon get stuck in the labyrinth the Frenchmen havecreated. Without monsters or anything to interact with,every tunnel looks just like every other, and I can’t find myway out.

One of the Frenchmen takes over (the one who made themaze) and he, too, soon gets lost. Which leads to lots ofexcited yelling (in French) and general frustration.

I tune out. It doesn’t bother me that they built a mazeso confusing they get lost in it themselves. In fact, that’skind of exciting.

But I can’t help wondering, if they get this game of theirs toactually work, will there be any other experience in theworld like it? Could a game made by a handful of Frenchgame designs students, working out of a tiny corner of atechnology company in Sweden, change the world?

Game Development of the labyrinth in Glimpse of Fear




I’m generally skeptical about such things, But on this day,after using this piece of technology, it all seems too possible.

. . .

* You can now get the Tobii EyeX here*Are you a #gamedev?Join the$1M funding program here.

. . .


http://www.tobii.com/boost?utm_source=medium&utm_medium=blog&utm_campaign=russmedium

http://bit.ly/1MbM9w4

https://twitter.com/tobiieyex



https://www.facebook.com/tobiiexperience

How Eye Tracking Will TOTALLY Change the Way You Game

Documents

Transcript of How Eye Tracking Will TOTALLY Change the Way You Game