70 PERVASIVEcomputing Published by the IEEE CS and IEEE ComSoc ■ 1536-1268/06/$20.00 © 2006 IEEE

T H E S M A R T P H O N E

The Smart Phone: AUbiquitous Input Device

Your mobile phone is the first trulypervasive computer. It’s always withyou, helping you keep in touch withothers and manage everyday tasks.Technological advances mean that

manufacturers can pack ever more features intothe small, convenient form factor.

Smart phones can already see, hear, and sensetheir environments. But, asMark Weiser pointed out, “Pro-totype tabs, pads, and boardsare just the beginning of ubiqui-tous computing. The real powerof the concept comes not fromany one of these devices; itemerges from the interaction ofall of them.”1 We’ll show howmodern mobile phones (Weiser’s

tabs) can interact with their environment, espe-cially large situated displays (Weiser’s boards).

Smart phones’ emerging capabilities are fuelinga rise in the use of mobile phones as input devicesto such resources as situated displays, vendingmachines, and home appliances. Mobile phones’prevalence gives them great potential to be thedefault physical interface for ubiquitous comput-ing applications. This would provide the founda-tion for new interaction paradigms, similar to howthe mouse and keyboard on desktop systems letthe GUI’s WIMP (windows, icons, menus, point-ers) paradigm emerge. However, realizing thispotential will require intuitive, efficient, and enjoy-able interaction techniques for applications in theubiquitous computing domain.

We survey interaction techniques that usemobile phones as input devices to ubiquitouscomputing environments, including two tech-niques we’ve developed. We use smart phone to

describe an enhanced mobile phone. Our analy-sis blurs the line between smart phones and PDAssuch as the Palm Pilot because the feature sets con-tinue to converge.

The input design spaceJames Foley, Victor Wallace, and Peggy Chan’s

classic article structures a taxonomy of inputdevices around the graphics subtasks that theycan perform: position, orient, select, path, quan-tify, and text entry.2 We use this taxonomy as aframework for our analysis of smart phones asubiquitous input devices. Although Foley and hiscolleagues performed their analysis in the desktopcomputing context, we can apply the subtasks toubiquitous computing. They naturally carry overto situated display interactions, but their applic-ability isn’t limited to graphical interactions.

PositionDuring a positioning task, you specify a posi-

tion in application coordinates. This subtaskoften serves as part of a command to place anentity at a particular position. Positioning tech-niques can be continuous, with the object posi-tion changing continually, or discrete, with theposition changing at the end of the task. Positioncan refer to screen position or to physical posi-tion in the real world. For example, you canadjust the height of motorized window blindsusing the position subtask.

The mobile phone is useful for positioningtasks in various ways.

Continuous indirect translation with a track pad.Remote commander is an application that lets youuse a PDA touch screen as a track pad to controla cursor’s relative position on a remote display.3

Feature-packed mobile phones might become the default physicalinterface for ubiquitous computing applications, providing thefoundation for new interaction paradigms.

Rafael Ballagas and Jan BorchersRWTH Aachen University

Michael RohsSwiss Federal Institute of Technology

Jennifer G. SheridanLancaster University

Continuous indirect translation withvelocity-controlled joysticks. A return-to-zero joystick controls a continuouslyrepositioned object’s velocity. Zero dis-placement of the joystick corresponds tozero velocity. Miika Silfverberg, ScottMacKenzie, and Tatu Kauppinen per-formed an in-depth study of isometric joy-sticks on handheld devices to control thecursor on public displays.4 Many mobilephones ship with binary-resolution, four-to eight-direction, return-to-zero joysticks.

Continuous indirect translation withaccelerometers. Accelerometers are begin-ning to emerge in handheld devices. Sam-sung’s SCH-S310 mobile phone, forexample, includes an integrated 3Daccelerometer. Rock ’n Scroll lets usersscroll (for example, through an electronicphoto album) by tilting the device.5

Although the Rock ’n Scroll technique isdesigned for interacting with an applica-tion directly on the device, we couldclearly extend it to positioning tasks inubiquitous computing environments.

Continuous indirect translation withdirectional step keys. Users control anobject’s location using up, down, left, andright step keys in 2D applications, plusin and out in 3D applications. In theBlinkenlights project (www.blinkenlights.de), users played the arcade classic Pongusing the side of a building as a display.Each window equaled one pixel on the18 � 8-pixel display (see figure 1). Play-ers connected to the display by calling adesignated phone number. Pressing 5 onthe phone keypad moves the paddle up;pressing 8 moves it down.

Continuous direct translation with cam-era tracking. Anil Madhavapeddy andhis colleagues6 present camera-basedinteractions that involve tagging inter-active GUI elements such as sliders anddials (see figure 2). By manipulating thephone camera’s position and orientation,you can position a graphical slider or ori-ent a graphical dial. This technique isanalogous to the classic light pen with atracking cross. As the light pen moves toa new position, the cross follows the pen’smotions. You can lose tracking by mov-ing the pen too fast, but you can easilyresume tracking by repositioning the penat the tracking cross. Madhavapeddy andhis colleagues’ interactions rely on atagged GUI widget instead of a cross fortracking.

In Madhavapeddy and his colleagues’positioning technique, the phone isresponsible for tracking, and the envi-

ronment is responsible for displayingand refreshing the tracked images. Otherresearchers have created interactionswith these roles swapped. For example,Shwetak Patel and Gregory Abowd aug-mented smart phones with laser point-ers,7 making them suitable for the posi-tioning tasks Dan Olsen and TravisNielsen describe,8 which track the laserusing a camera in the environment.

Continuous indirect translation with cam-era tracking. The C-Blink system lets youposition a cursor on a large display usinga mobile phone with a colored screen, asfigure 3 shows.9 You run a program on

JANUARY–MARCH 2006 PERVASIVEcomputing 71

Figure 2. Using a phone to manipulatetagged widgets such as dials and sliders.(figure courtesy of Anil Madhavapeddyand his colleagues)

Figure 3. A user waves the phone screenin front of a camera to control cursorposition. (figure courtesy of KentoMiyaoku and colleagues)

Figure 1. View of the side of the buildingused to play Pong, with buttons on themobile phone controlling the paddle.(figure courtesy of Blinkenlights)

Camera

Cursor

the phone that rapidly changes thescreen’s hue and then wave the phone infront of a camera mounted above the dis-play. The displayed hue sequence encodesan ID to support multiple users. The cam-era tracks this signal’s absolute positionin the camera image to control the cur-sor on the display.

Discrete direct translation using a cameraimage. Figure 4 illustrates the Point &Shoot interaction technique,10 in whichthe phone screen is the locus of atten-

tion. You aim at the desired positionusing a cross hair displayed over a livecamera image on the mobile phone (seefigure 4a). To reposition the cursor tothis new target, you press and release thejoystick button. (An alternative imple-mentation of the Point & Shoot tech-nique uses pen input instead of the cross-hair image. You reposition the cursor byselecting the desired position directly onthe live camera image displayed on thetouch-sensitive phone screen.) This trig-gers a grid of tags for the camera to

briefly superimpose over the large dis-play contents, as figure 4b shows. Point& Shoot uses the grid to derive a per-spective-independent coordinate systemand determines the selected point withpixel-level accuracy, which the visualcode tags’ properties allow.11

Analogous to this system is one varia-tion of the classic light pen that discretelydetermines position by displaying a rasterscan when you click a button on the pen.When the pen senses the raster scan, thesystem knows the pen’s position becauseof a tight coupling between the pen anddisplay clocks. In Point & Shoot, a visualtag grid replaces the raster scan’s func-tionality. This technique is robust to dif-ferent display technologies and achievesloose coupling between camera and dis-play. Point & Shoot needs only onevisual tag in the camera view to estab-lish a coordinate system, but using thegrid increases the probability of satisfy-ing this requirement.

Continuous indirect translation withcamera optical flow. The Sweep interac-tion technique uses optical-flow imageprocessing (see figure 5), which involves

72 PERVASIVEcomputing www.computer.org/pervasive

T H E S M A R T P H O N E

Figure 4. Point & Shoot interaction. (a) A user aims the phone screen at a puzzle piece on a large situated display. (b) Pressing thejoystick indicates a selection. The phone sends a command to the large display to briefly superimpose a visual code grid over thedisplay contents. The phone uses the code grid to compute the target coordinates in the captured image. (c) The grid disappearsand the large display highlights the target puzzle piece, indicating successful selection.

Figure 5. The Sweep technique uses camera input and simplified optical flowimage processing to control a cursor.

(a) (b) (c)

“Click”

rapidly sampling successive images froma camera phone and sequentially com-paring them to determine relative motionin the (x, y, �) dimensions.10 The tech-nique requires no visual tags. You don’teven need to point the camera at the dis-play. To invoke the Sweep function, yousteer the joystick downward. This actsas a clutch to indicate to the system thatyou’re actively controlling the cursor.Pressing the joystick button indicatesselection or dragging. Waving the phonein the air controls the cursor. You canrelease the clutch button to repositionyour arm, similar to how you lift amouse to reposition it on the desktopsurface. In Sweep mode, you can ignorethe phone screen and observe cursormovement on the display. (See also the“Designing for Serendipity” sidebar.)

OrientThe orienting subtask involves speci-

fying an orientation (not a position) in acoordinate system. Like position, orien-tation isn’t limited to graphics subtasksbecause it can relate to physical orienta-tion in the real world. For example, youcould use the orientation subtask to con-trol the orientation of a security camera,spotlight, or steerable projector.

As with position, the orientation tech-niques can be continuous or discrete.Some of Foley and his colleagues’ origi-nal graphics interactions carry overdirectly for smart phones—for example,indirect continuous orientation with avelocity-controlled joystick and discreteorientation with angle type-in. We sur-vey the remaining techniques.

Continuous indirect orientation with loca-tor devices. You can specify the orienta-tion angle using a continuous quantifieror one axis of a physical locator. TheSweep technique supports detection ofrotation around the z axis (coming outof the display), allowing interactions suchas rotating a puzzle piece in a jigsaw puz-

zle application, where you use the phonelike a ratchet to adjust orientation.Sweep’s optical-flow processing can alsodetect rotation around the x and y axes.However, Sweep maps rotation aroundthe y axis to translate position along thex axis and maps rotation around the xaxis to translate position along the y axis.

Continuous direct orientation with cam-era tracking. You can orient Mad-havapeddy’s tagged GUI dials using thephone camera to track movement.

Discrete direct orientation with cameraimage. You can use the Point & Shoottechnique to specify orientation by rotat-ing the camera around the z axis.

SelectIn the selection subtask, you select

from a set of alternatives, such as a setof commands. A common input optionis to make a menu selection using a cur-sor controlled through the positioningsubtask. In graphics interactions, thesecan be displayed entities, such as graph-ical icons that form part of the GUI com-position. However, this subtask is alsorelevant to ubiquitous computing inter-actions because you can select a physi-cal object to operate upon, such as select-ing a lamp to adjust its setting.

Many selection techniques carry overdirectly from Foley and his colleagues’analysis, such as simulated pick with cur-sor match, typing in character stringname for command prompts, and push-ing soft key buttons, in which commandsare presented as labeled buttons. The fol-lowing sections describe the remainingselection techniques.

Direct pick of tagged objects. XeroxPARC’s E-tag project investigates howtagged objects can present informationon a wireless mobile computer equippedwith an electronic tag reader.12 Forexample, selecting a book by scanningits embedded radiofrequency identifier(RFID) tag would bring up a Web refer-ence to the book (perhaps letting youpurchase it). Developers have proposedsimilar interactions for visual tags in theenvironment11 and tagged GUI ele-ments6,13 in which you use a camera toacquire an image to decode the selectedtag. Patel and Abowd7 present a physi-cal-world selection method for mobilephones in which a modulated laserpointer signal triggers a photosensitivetag placed in the environment, lettingyou bring up a menu to control theobject on your handheld device.

With RFIG lamps,14 you use a hand-held projector to select objects with pho-tosensitive RFID tags in the physicalworld. The handheld projector emits agray code pattern with which the tagsdetermine their relative positions in theprojected view. By waving the handheldprojector, you can navigate a cursor inthe center of the projected view to selectindividual objects.

Direct pick with camera. You can use thePoint & Shoot technique without a cur-sor to select items on a display or in thereal world, such as printed selections ona poster.

Direct pick by laser pointer. Brad Myersand his colleagues15 propose semanticsnarfing, a multidevice technique thatuses a laser pointer to make a coarse-

JANUARY–MARCH 2006 PERVASIVEcomputing 73

A key design consideration is the ease and speed of setting up a data connection

between the phone and the environment or the device it’s controlling. For

projects that use short-range wireless communications models such as Bluetooth, you

can use visual or RFID tags to encode the environment’s connection information. This

creates a low threshold of use and allows for highly serendipitous interactions.

Designing for Serendipity

74 PERVASIVEcomputing www.computer.org/pervasive

T H E S M A R T P H O N E

grained selection of a screen region on adisplay in a camera-monitored environ-ment. The interaction then composes aGUI on the handheld device’s screen forthe fine-grained GUI selection.

Voice recognition. You speak the selectedcommand’s name, and a speech recog-nizer determines what command wasspoken. The Personal Universal Con-troller supports automatic generation ofspeech interfaces (as well as graphicalinterfaces) to issue commands to real-world objects.16

Gesture recognition. You can make asequence of movements using a contin-uous positioning device such as a joy-stick, camera, track pad, or accelerome-ter. Shwetak Patel, Jeffrey Pierce, andGregory Abowd use gesture recognitionof a handheld device’s accelerometerdata to authenticate users wanting toaccess data on their mobile phonesthrough an untrusted public terminal.17

The authors describe how to use thistechnology to securely bring up data onthe public terminal from their phonewithout removing the phone from theirpocket or bag.

PathPathing involves specifying a series of

positions and orientations over time, soit’s always continuous and closed loop.Although this task’s requirements differfrom those of position and orientation,pathing adheres to the same taxonomyas the corresponding positioning and ori-enting techniques.

QuantifyQuantifying involves specifying a

value or number within a range of num-bers. In our survey, we performed thequantifying tasks using phone inputthrough a GUI using 1D position or ori-entation subtasks. You can also usetagged GUI elements for numeric inputwithin a bounded range.13

TextText entry for mobile phones is a well-

studied area because it’s central to text-based mobile communications (such asshort message service) and personal infor-mation management functionality. Moreon this topic is available elsewhere.18

Evaluation of techniquesThe use of mobile phones as input

devices for their environment is still new,and only a handful of thorough evalua-tions of the different techniques exist.4,10,15

These studies use different experimentalparameters, making the results difficultto directly compare. Table 1 compares thebest-studied task: positioning. To createthis table, we combined our knowledgeof the interaction techniques and the col-lective knowledge of their desktop com-puting counterparts to extrapolate roughestimates for various ergonomic mea-sures. We borrowed the ergonomic para-meters primarily from Foley and his col-leagues’ interaction technique survey:

• Perceptual, cognitive, and motor load.These ergonomic measures refer tobasic psychological processes that arenecessary for any interaction technique.

• Distance sensitivity. Because theamount or degree of separationbetween the display and the user isoften dynamic and unpredictable, therange of distances the interaction willsupport is an important design con-sideration. Both display size and inter-action distance influence perspectivesize. Distance sensitivity plays a sig-nificant role in determining what types

TABLE 1Comparison of mobile phone-based positioning techniques.

Project or author Technique Ergonomic measure*

Sweep10 Indirect, relative camera M M H M H M H M L C

Miika Silfverberg and colleagues4 Indirect, relative H M L M M M L M L Cvelocity-controlled joystick

Remote commander3 Indirect, relative track pad M M M M H M M M L C

Rock ’n Scroll5 Indirect, relative accelerometer H M L M M M L M L C

Blinkenlights (www.blinkenlights.de) Indirect, relative directional step keys H H L H M M L M L D

Point & Shoot10 Direct, absolute camera M H M H H M M H H D

C-Blink9 Indirect, absolute camera M M H M M M H M M C

Anil Madhavapeddy and colleagues6 Direct, relative camera L M M M M M M L H C

Dan Olsen and Travis Nielsen8 Direct, absolute laser pointer L M M M M M M H H D

* L = low, M = medium, H = high, C = continuous, D = discreteC

og

nit

ive

load

Per

cep

tual

load

Vis

ual

acq

uis

itio

n

Moto

r lo

ad

Moto

r ac

quis

itio

n

Ease

of

lear

nin

g

Fati

gue

Erro

r p

ron

enes

s

Dis

tan

ce s

ensi

tivi

ty

Feed

bac

k

of interaction techniques suit a par-ticular task. For example, interactionsthat require aiming, such as Point &Shoot, work best when you’re near thetarget, making the target perceptivelylarger. On the other hand, distancerarely significantly affects interactionsthat don’t require aiming, such asthose using the Sweep technique.

The remaining measures are selfexplanatory, but further clarification isavailable elsewhere.2

Spatial layout of design spaceThe Foley-style graph in figure 6 sum-

marizes the position, orient, and selecttechniques. Stuart Card, Jock Mackin-lay, and George Robertson point out thatthis format is somewhat ad hoc and thatthe Foley taxonomy has not tried todefine a notion of completeness.19 Cardand his colleagues then built on WilliamBuxton’s work20 to create a systematicspatial layout of input devices’ designspace. This design space is well-knownand effectively captures manual devices’physical properties. However, it fails tocapture many traits that are relevant toubicomp interactions, such as modalityor feedback, as we discuss elsewhere.21

Using Foley and his colleagues’ tax-onomy, we propose a new five-part spa-tial layout for the mobile phone interac-tion tasks we discuss in this survey:

• dimensionality,• relative versus absolute measurements,• interaction style (direct versus indirect), • supported subtasks (position, orient,

and selection), and • feedback from the environment (con-

tinuous versus discrete).

Figure 7 shows the resulting catego-rization. Using this graphical layout, wecan pinpoint gaps in the breadth of theinteraction techniques surveyed. This

layout isn’t comprehensive, and otherlayouts might pinpoint further gaps. Analternative layout might include phone

output modalities used, which woulddemonstrate these techniques’ sparse useof auditory and haptic feedback.

JANUARY–MARCH 2006 PERVASIVEcomputing 75

Orient

O 1

O 2

Direct withpick device

O 1.1 (P ) Camera [Point & Shoot]O 1.2 (P ) Camera tag tracking [Madhavapeddy]

Indirect withlocator device

O 2.1 (P ) Camera [Sweep](P ) Joystick (velocity)O 2.2

(P ) = on phone (E ) = in environment

O 3 (Text input)

Position

P 1

P 3

P 4

Direct withpick device

P 1.1(P ) Camera + (P ) On-screen cursor [Point & Shoot10]

P 1.2 (P ) Camera + (P) touch screenP 1.3 (P ) Camera tag tracking [Madhavapeddy6]

Indirect withlocator device

P 3.1 (P ) Camera [Sweep10]P 3.2

(P ) Joystick (velocity) [Silfverberg4]P 3.3P 3.4 (P ) Trackpad [Remote commander3]

(P ) Display + (E ) Stationary camera [C-Blink9]

Indirect withdirectionalcommand

P 4.1 (P ) Directional step keys [Blinkenlights]

P 3.5 (P ) Cursor control keys with auto-repeat

P 1.4(E ) Camera + (P ) Laser pointer [Olsen8]

P 2 With numericalcoordinates

With numericalcoordinates

(Text input)

(See selection)

Selection

S 1

S 3

S 4

S 6

S 2

Of object with directpick device

S 1.1(P ) Camera + (P ) On-screen cursor [Point & Shoot]

S 1.5 (P ) Camera + (E ) visual tags [Madhavapeddy]

S 1.2 (P ) Camera + (P ) pen input

S 1.3(E ) Light sensor (for example, camera) + (P ) Laser pointer [Olsen, semantic snarfing15, Patel7]

S 1.6 (P ) RFID reader + (E ) RFID tags [Want]

Indirect withcursor match

S 3.1 (P ) Camera [Sweep]S 3.2

(P ) Joystick (velocity) [Silfverberg]S 3.3S 3.4 (P ) Trackpad [Remote commander]

With characterstring name (Text input)

Gesturerecognition

S 1.4(P ) Handhjeld projector + (P ) RFID reader +(E ) Light sensitive RFID tags [RFIG14]

(P ) Display + (E ) stationary camers [C-Blink]

S 6.1 (P ) CameraS 6.2 (P ) Display + (E ) cameraS 6.3 (P ) Accelerometer [Patel]

Voice input S 2.1 (E ) Voice recognizer + (P ) microphone [PUC16]

S 3.5 (P ) Cursor control keys

S 5 Button push S 5.1 (P ) Programmed function keyboardS 5.2 (P ) Soft keys

S 6.4 (P ) Pen input

P 3.6 (P ) Accelerometer [Rock 'n Scroll5]

S 3.6 (P ) Accelerometer [Rock 'n Scroll]

Figure 6. Summary of position, orientation, and selection techniquesusing a smart phone as an input device.

Examining these new mobilephone interaction techniqueswithin the structure of a taxon-omy and design space helps

designers understand the relationshipsbetween the different techniques andidentify which techniques are mostsuited for a particular interaction sce-nario. The new physical layout helpsreveal gaps in the design space of inputdevices for ubiquitous computing sce-narios. These gaps indicate areas wherefurther research and innovation arerequired for new mobile phone interac-tion techniques. We plan to continue to

work towards a comprehensive designspace for ubiquitous computing interac-tions to help identify areas of futureresearch.

REFERENCES1. M. Weiser, “The Computer for the 21st

Century,” Scientific Am., no. 265, Sept.1991, pp. 94–104.

2. J.D. Foley, V.L. Wallace, and P. Chan, “TheHuman Factors of Computer GraphicsInteraction Techniques,” IEEE ComputerGraphics and Applications, vol. 4, no. 11,1984, pp. 13–48.

3. B.A. Myers, H. Stiel, and R. Gargiulo, “Col-

laboration Using Multiple PDAs Connectedto a PC,” Proc. 1998 ACM Conf. Computer-Supported Collaborative Work (CSCW 98),ACM Press, 1998, pp. 285–294.

4. M. Silfverberg, I.S. MacKenzie, and T.Kauppinen, “An Isometric Joystick as aPointing Device for Handheld InformationTerminals,” Proc. Graphics Interface, 2001,pp. 119–126.

5. J.F. Bartlett, “Rock ’n Scroll is Here toStay,” IEEE Computer Graphics and Appli-cations, vol. 20, no. 3, 2000, pp. 40–45.

6. A. Madhavapeddy et al., “Using CameraPhones to Enhance Human-ComputerInteraction,” Adjunct Proc. 6th Int’l Conf.Ubiquitous Computing (UbiComp 2004),2004, http://ubicomp.org/ubicomp2004/adjunct/demos/madhavapeddy.pdf.

7. S.N. Patel and G.D. Abowd, “A 2-WayLaser-Assisted Selection Scheme for Hand-helds in a Physical Environment,” Proc. 5thInt’l Conf. Ubiquitous Computing (Ubi-Comp 03), LNCS 2864, A.K. Dey, A.Schmidt, and J.F. McCarthy, eds., Springer,2003, pp. 200–207.

8. D.R. Olsen and T. Nielsen, “Laser PointerInteraction,” Proc. SIGCHI Conf. HumanFactors in Computing Systems (CHI 01),ACM Press, 2001, pp. 17–22.

9. K. Miyaoku, S. Higashino, and Y. Tono-mura, “C-Blink: A Hue-Difference-BasedLight Signal Marker for Large Screen Inter-action via any Mobile Terminal,” Proc.17th Ann. ACM Symp. User Interface Soft-ware and Technology (UIST 04), ACMPress, 2004, pp. 147–156.

10. R. Ballagas, M. Rohs, and J.G. Sheridan,“Sweep and Point and Shoot: Phonecam-Based Interactions for Large Public Dis-plays,” CHI 05 Extended Abstracts HumanFactors in Computing Systems, ACM Press,2005, pp. 1200–1203.

11. M. Rohs, “Real-World Interaction withCamera Phones,” Proc. 2nd Int’l Symp.Ubiquitous Computing Systems (UCS 04),LNCS 3598, Springer-Verlag, 2005, pp.74–89.

12. R. Want et al., “Bridging Physical and Vir-tual Worlds with Electronic Tags,” Proc.

76 PERVASIVEcomputing www.computer.org/pervasive

T H E S M A R T P H O N E

4

5 6

10

11 1312

14

1516

1. Trackpad Remote Commander3

2. Joystick (velocity) Romeo 3. Accelerometers Rock ‘n Scroll5 4. Directional step keys Blinkenlights 5. Camera Madhavapeddy (Slider)6

6. Camera Madhavapeddy (Dial)6

7. (E ) Camera + (P ) Display C-Blink9

8. (P ) Laser + (E ) Camera Olsen8

9. Camera Point & Shoot10

10. Camera Sweep10 11. (P ) RFID reader + (E ) Tags E-tag12

12. (P ) Camera + (E ) Tags Madhavapeddy6

13. (P ) Microphone + (E ) Voice recognition Personal universal controller16

14. (P ) Projector + (P ) RFID reader + (E ) Photo sensitive tags RFIG14

15. (P ) Laser + (E ) Camera Semantic snarfing15

16. (P ) Laser + (E ) Camera Patel7

Position

Orientation

Selection

Abso

lute

Rela

tive

DirectIndirect

DirectIndirect

1D 2D 3D

1 23

DimensionalityN/A

10

7

8

Continuous ContinuousDiscrete ContinuousDiscrete ContinuousDiscrete Discrete

9

Feedback from environment

9

Figure 7. Classification of mobile phoneinteractions that the projects surveyedhave implemented. Inspecting thediagram reveals opportunities for futurework—for instance, developinginteraction techniques that support 3Dorientation or 3D positioning.

SIGCHI Conf. Human Factors in Comput-ing Systems (CHI 99), ACM Press, 1999,pp. 370–377.

13. M. Rohs, “Visual Code Widgets forMarker-Based Interaction,” Proc. 25thIEEE Int’l Conf. Distributed ComputingSystems (ICDCS Workshops), Int’l Work-shop Smart Appliances and Wearable Com-puting (IWSAWC), IEEE CS Press, 2005,pp. 506–513.

14. R. Raskar et al., “RFIG Lamps: Interactingwith a Self-Describing World via Photo-sensing Wireless Tags and Projectors,”Trans. Graphics, vol. 23, no. 3, 2004, pp.406–415.

15. B.A. Myers et al., “Interacting at a Distance:Measuring the Performance of Laser Point-ers and Other Devices,” Proc. SIGCHI Conf.Human Factors in Computing Systems(CHI 02), ACM Press, 2002, pp. 33–40.

16. J. Nichols et al., “Personal Universal Con-trollers: Controlling Complex Applianceswith GUIs and Speech,” Proc. SIGCHI Conf.Human Factors in Computing Systems(CHI 03), ACM Press, 2003, pp. 624–625.

17. S.N. Patel, J.S. Pierce, and G.D. Abowd, “AGesture-Based Authentication Scheme forUntrusted Public Terminals,” Proc. 17thAnn. ACM Symp. User Interface Softwareand Technology (UIST 04), ACM Press,2004, pp. 157–160.

18. L. Butts and A. Cockburn, “An Evaluationof Mobile Phone Text Input Methods,”Proc. Conf. Research and Practice in Infor-mation Technology Series, 3rd AustralasianConf. User Interfaces, vol. 7, AustralianComputer Soc., 2002, pp. 55–59.

19. S.K. Card, J.D. Mackinlay, and G.G.Robertson, “A Morphological Analysis ofthe Design Space of Input Devices,” Trans.Information Systems, vol. 9, no. 2, 1991,pp. 99–122.

20. W. Buxton, “Lexical and Pragmatic Con-siderations of Input Structures,” SIGGRAPH

Computer Graphics, vol. 17, no. 1, 1983,pp. 31–37.

21. R. Ballagas et al., “iStuff: A Physical UserInterface Toolkit for Ubiquitous Comput-ing Environments,” Proc. SIGCHI Conf.Human Factors in Computing Systems(CHI 03), ACM Press, 2003, pp. 537–544.

For more information on this or any other comput-ing topic, please visit our Digital Library at www.computer.org/publications/dlib.

JANUARY–MARCH 2006 PERVASIVEcomputing 77

the AUTHORS

Rafael “Tico” Ballagas is a computer science doctoral candidate and research assis-tant at RWTH Aachen University in Aachen, Germany. His research interests includepostdesktop user interfaces, distributed I/O, physical toolkits, and infrastructuresolutions for ubiquitous computing environments. Ballagas has an MSc in electricalengineering from Stanford University. Contact him at Media Computing Group,RWTH Aachen Univ., 52056 Aachen, Germany; ballagas@cs.rwth-aachen.de.

Michael Rohs is a senior research scientist with Deutsche Telekom Laboratories at TUBerlin. His primary research interest is in mobile and pervasive user interfaces, includ-ing interfaces for handheld devices and large public displays and the integration ofphysical and virtual aspects of the user’s current environment. Rohs developed avisual code system for camera phones that enables gesture-based interactions withpassive media as well as electronic displays. He has a PhD in computer science fromETH Zurich, Switzerland, where he was a research assistant at the Institute for Perva-sive Computing. Contact him at Deutsche Telekom Laboratories, Ernst-Reuter-Platz 7,10587 Berlin, Germany; michael.rohs@telekom.de.

Jennifer G. Sheridan is a research associate in Lancaster University’s ComputingDepartment and the director of .:thePooch:., the University installation and perfor-mance arts group that has established a national reputation for its work on theboundaries of arts and technology. She’s also part of Equator, a research consortiumsupported by the UK’s Engineering and Physical Sciences Research Council. Herresearch interest is exploring the types of performative interactions that occur withnon-task-based interactive uses of technologies in performance and installation artsparticularly for playful arenas and unanticipated performance spaces. Sheridan has

conducted several new media happenings and extreme prototyping events and recently completed aPhD in computer science from Lancaster University. Contact her at InfoLab21, South Dr., ComputingDept., Lancaster Univ., Lancaster UK, LA1 4WA; sheridaj@comp.lancs.ac.uk.

Jan Borchers is a professor of computer science and head of the Media ComputingGroup at RWTH Aachen University. With his research group, he explores the field ofhuman-computer interaction, with a particular interest in new postdesktop user inter-faces for smart environments, ubiquitous computing, interactive exhibits, and time-based media such as audio and video. Borchers has a PhD in computer science fromDarmstadt University of Technology. He’s a member of the ACM, SIGCHI, BayCHI, andGI (the German Informatics Society). Contact him at Media Computing Group,RWTH Aachen Univ., 52062 Aachen, Germany; borchers@cs.rwth-aachen.de.

www.computer.org/join/grades.htm

GIVE YOUR CAREER A BOOST ■ UPGRADE YOUR MEMBERSHIP

Advancing in the IEEE Computer Society can elevate your standing in the profession.

Application to Senior-grade membership recognizes

✔ ten years or more of professional expertise

Nomination to Fellow-grade membership recognizes

✔ exemplary accomplishments in computer engineering

REACH HIGHER