Mobile Devices for Control of Ubiquitous Multimedia Brad A. Myers [email protected] Brad A. Myers...

Mobile Devices for Control ofUbiquitous Multimedia


Brad A. Myers

[email protected]

http://www.pebbles.hcii.cmu.edu/

Brad A. Myers

[email protected]


Human Computer Interaction Institute Human Computer Interaction Institute

School of Computer ScienceSchool of Computer Science

Carnegie Mellon UniversityCarnegie Mellon University

Brad Myers Carnegie Mellon 2

Handheld Devices Typically, handheld devices for:

Communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement

“PocketPC” As a media player

Handheld Devices for Remote Control of other devices and media Remote Control of PCs Remote Control of appliances

Typically, handheld devices for: Communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement

“PocketPC” As a media player

Handheld Devices for Remote Control of other devices and media Remote Control of PCs Remote Control of appliances


Problem

Appliances are too complex Appliances are too complex


Problem

Too many remotes Too many remotes


Problem

April 29, 1991 April 29, 1991


Approach

Use a personal handheld device as an Interface to the PC and to the Appliances

Use a personal handheld device as an Interface to the PC and to the Appliances


History of Communication

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –ActiveSync once a day

ParcTab – continuous communication

Early “organizers” – no communication

Newton – extra cost for synchronization

Palm – HotSync once a day

WindowsCE/PocketPC –ActiveSync once a day


Now, Handhelds are Communicating

802.11

BlueTooth

Cell-phone network (G3)

802.11

BlueTooth

Cell-phone network (G3)

Toshiba e740

HP iPaq 3870

Microsoft SmartPhone


Pebbles Project

Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the

same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances

Multiple users with their devices Single user with multiple devices

Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the

same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances

Multiple users with their devices Single user with multiple devices


Pebbles is:

PEBBLES

PEBBLES

DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.

DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.

http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/


Pebbles for MultiMedia

How will handheld devices fit into the user’s overall media space? Where does media come from? Where is the media is displayed? How is the media controlled?

How can handheld devices improve the user interfaces of other media devices, rather than just being another gadget to master?

How will handheld devices fit into the user’s overall media space? Where does media come from? Where is the media is displayed? How is the media controlled?

How can handheld devices improve the user interfaces of other media devices, rather than just being another gadget to master?

Controlling a PCControlling a PC

Remote Commander Slide Show Commander Command Post of the Future Shortcutter

Remote Commander Slide Show Commander Command Post of the Future Shortcutter

http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/


Remote Commander

Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and

grab mouse All mouse and keyboard functions

Use PDA like touchpad Graffiti or our own pop-up

keyboard Word prediction and completion

Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and

grab mouse All mouse and keyboard functions

Use PDA like touchpad Graffiti or our own pop-up

keyboard Word prediction and completion


PocketPC version Get PC’s screen onto PocketPC Full view, or one-to-one zooming

Scroll with iPaq’s buttons Control or scribble

Get PC’s screen onto PocketPC Full view, or one-to-one zooming

Scroll with iPaq’s buttons Control or scribble


For People with Muscular Disabilities

Using handhelds as interface to PCs People with Muscular Dystrophy

have fine-motor control but losegross motor control

Difficulties with mouse and keyboard,but stylus OK

So use Remote Commanderas PC’s keyboard and mouse

Using handhelds as interface to PCs People with Muscular Dystrophy

have fine-motor control but losegross motor control

Difficulties with mouse and keyboard,but stylus OK

So use Remote Commanderas PC’s keyboard and mouse


Alphabet for Accurate Gestures Use a square hole in a template to increase

accuracy Just as easy to learn

as Graffiti™ 18% more accurate

for able novices (p<.02)

Patent Pending Also works with joystick

Use a square hole in a template to increase accuracy Just as easy to learn

as Graffiti™ 18% more accurate

for able novices (p<.02)

Patent Pending Also works with joystick


Power Point Control Use PC to give the presentation Use hand-held to control the PC

Two-way communication Hand-held shows picture of slide,

notes, timer

Use PC to give the presentation Use hand-held to control the PC

Two-way communication Hand-held shows picture of slide,

notes, timer


SlideShow Commander, cont.

See list of slide titles Scribble Tap on on-screen buttons

and hyper-links Preview other

slides Control

demonstrations

See list of slide titles Scribble Tap on on-screen buttons

and hyper-links Preview other

slides Control

demonstrations


Control at a Distance

Controlling a PC fromacross the room Meetings, etc.

Use a laser pointer? Studies

Controlling a PC fromacross the room Meetings, etc.

Use a laser pointer? Studies


Command Post of the Future Share media & control across multiple devices

Big displays, speech, gesture And handhelds

Share media & control across multiple devices Big displays, speech, gesture And handhelds


CPOF, cont.

Provides remote control of main PC battleboard from handhelds

Also, private drill-down of public information Small versions of map and data

visualizations on handhelds Interact locally, and display

changes and annotationswhen ready

Provides remote control of main PC battleboard from handhelds

Also, private drill-down of public information Small versions of map and data

visualizations on handhelds Interact locally, and display

changes and annotationswhen ready


Shortcutter

User-created panels of controls Create custom interfaces and

extensions to PC applications And then take them with you

Direct manipulationfor edit, then setproperties

PocketPC or Palm

User-created panels of controls Create custom interfaces and

extensions to PC applications And then take them with you

Direct manipulationfor edit, then setproperties

PocketPC or Palm


Shortcutter Widgets

Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel

Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel


Scenarios of Use

Lean-back mail reading

Controlling media players

… and many others

Lean-back mail reading

Controlling media players

… and many others

Controlling AppliancesControlling Appliances

PhD research of Jeffrey Nichols Assisted by undergrads and MS students

PhD research of Jeffrey Nichols Assisted by undergrads and MS students

http://www.pebbles.hcii.edu/puc/http://www.pebbles.hcii.edu/puc/

Problem

Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock

Increasingly computerized Low usability

Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock

Increasingly computerized Low usability


Existing “universal” controls

Pre-programmed at the factory with a subset, Or, laboriously hand-programmed by the user

Pre-programmed at the factory with a subset, Or, laboriously hand-programmed by the user


Our Approach Handheld is a “Personal Universal Controller” (PUC) Two-way communication

Appliance describes its functions Handheld PUC

Automatically creates interface Controls the appliance Displays feedback about appliance status

Handheld is a “Personal Universal Controller” (PUC) Two-way communication

Appliance describes its functions Handheld PUC

Automatically creates interface Controls the appliance Displays feedback about appliance status

Specifications

Control

Feedback


Automatic Generation of UIs

Benefits All interfaces consistent for a user

With conventions of the handheld Even from multiple manufacturers

Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)

A Hard Problem Previous automatic systems have not

generated high quality interfaces

Benefits All interfaces consistent for a user

With conventions of the handheld Even from multiple manufacturers

Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)

A Hard Problem Previous automatic systems have not

generated high quality interfaces


Research Challenges

1. Automatic Design of Graphical User Interfaces

2. Automatic Design of Speech User Interfaces

3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software

1. Automatic Design of Graphical User Interfaces

2. Automatic Design of Speech User Interfaces

3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software


Hand-Generated Graphical Interfaces

First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering

Machine

Used Embedded Visual Basic

Ensured quality with heuristicanalysis and think-aloud studies

Compared with manufacturer’sinterfaces

First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering

Machine

Used Embedded Visual Basic

Ensured quality with heuristicanalysis and think-aloud studies

Compared with manufacturer’sinterfaces


Results of Comparison

Using PUC, users took 50% less time & made 50% fewer errors

All differences are significant (p < 0.05)

Using PUC, users took 50% less time & made 50% fewer errors

All differences are significant (p < 0.05)


Current PUC Specification Language

XML Full documentation for the

specification language and protocol:

http://www.pebbles.hcii.cmu.edu/puc/

Contains sample specification for a stereo

XML Full documentation for the

specification language and protocol:

http://www.pebbles.hcii.cmu.edu/puc/

Contains sample specification for a stereo


Properties of PUC Language

State variables & commands Each can have multiple labels

Useful when not enough room Typed variables

Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.

Optional labels for values Hierarchical Structure

Groups

State variables & commands Each can have multiple labels

Useful when not enough room Typed variables

Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.

Optional labels for values Hierarchical Structure

Groups


Dependency Information

Crucial for high-quality interfaces Expressed as <active-if> clauses

Operations: Equals, Less-Than,

Greater-Than Combined Logically

AND, OR Used for:

Dynamic graying out Layout Widget selection

Crucial for high-quality interfaces Expressed as <active-if> clauses

Operations: Equals, Less-Than,

Greater-Than Combined Logically

AND, OR Used for:

Dynamic graying out Layout Widget selection


Specifications

Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator Parts of GMC Yukon Denali SUV Etc.

Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator Parts of GMC Yukon Denali SUV Etc.


Controller Generators

iPaq PocketPC

SmartPhone No touchscreen

Desktop (TabletPC)

Speech

iPaq PocketPC

SmartPhone No touchscreen

Desktop (TabletPC)

Speech


Examples of Generated GUIs

Shelf stereo Windows Media Player

Shelf stereo Windows Media Player


Generating Speech Interfaces “Universal Speech Interface” (USI) project

Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi

Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning

Will use dependency information to help with disambiguation and explanation

Supports queries and spoken feedback Paraphrases as confirmation

“Universal Speech Interface” (USI) project Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi

Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning

Will use dependency information to help with disambiguation and explanation

Supports queries and spoken feedback Paraphrases as confirmation


Adaptors

“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly

Custom hardware Custom software

Lutron Windows Media Player

X-10 Light switches, etc.

“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly

Custom hardware Custom software

Lutron Windows Media Player

X-10 Light switches, etc.

AV/C (standard protocol) Sony CamCorder

HAVi UPnP

Axis Camera

AV/C (standard protocol) Sony CamCorder

HAVi UPnP

Axis Camera


Demonstration

Sony CamCorder Windows Media Player Image library Two way communication:

When state changes from application or appliance, GUI is updated

Can query state with speech

Sony CamCorder Windows Media Player Image library Two way communication:

When state changes from application or appliance, GUI is updated

Can query state with speech


Current Work on PUC Controlling User “Experiences”

One command for multiple devices Play a DVD sets DVD, cable, TV Setting alarm clock sets heat, coffee, car

Consistency for Users And user preferences

More devices UPnP Etc.

Controlling User “Experiences” One command for multiple devices Play a DVD sets DVD, cable, TV Setting alarm clock sets heat, coffee, car

Consistency for Users And user preferences

More devices UPnP Etc.


For More Information

Many papers on the Pebbles web site On PUC and other work

Most programs available for free downloading:

SlideShow Commander islicensed for commercial sale See handango.com

Many papers on the Pebbles web site On PUC and other work

Most programs available for free downloading:

SlideShow Commander islicensed for commercial sale See handango.com

http://www.cs.cmu.edu/~pebbleshttp://www.cs.cmu.edu/~pebbles


Conclusions

As more and more electronics are computerized and communicating, mobile devices can help control them

Handheld devices can improve the user interfaces of everything else Not just be another gadget to be learned

As more and more electronics are computerized and communicating, mobile devices can help control them

Handheld devices can improve the user interfaces of everything else Not just be another gadget to be learned


Thanks to Pebbles Students This research performed by a large number

of students since 1997:

This research performed by a large number of students since 1997:

Htet Htet AungRishi BhatnagarBen Bostwick Franklin ChenYu Shan ChuangKaren CrossCarl EvankovichIvan GonzalezMarc KhadpeDave KongChun-Kwok LeeJoonhwan Lee

Jennifer LiYuhua LiLeo LieJack LinKevin LitwackA. Chris LongColin McCabeRob MillerJeff NicholsChoon Hong PeckMathilde PignolSuporn Pongnumkul

Brandon RothrockRajesh SeenichamyPegeen ShenHerbert StielJeff StylosClaire TokarMarsha Tjandra Adrienne WarmackJacob O. WobbrockJerry YangSunny YangBrian Yeung


Thanks to Our Sponsors!The Pebbles research is supported by grants from:

DARPANSFMicrosoftGeneral MotorsNEC FoundationPittsburgh Digital Greenhouse

And equipment grants from: Hewlett PackardLucent TechnologiesLantronix, Inc.Lutron, Inc.MitsubishiPalm Computing

The Pebbles research is supported by grants from:DARPANSFMicrosoftGeneral MotorsNEC FoundationPittsburgh Digital Greenhouse

And equipment grants from: Hewlett PackardLucent TechnologiesLantronix, Inc.Lutron, Inc.MitsubishiPalm Computing

Symbol TechnologiesIBMSMART TechnologiesVividLogicSynergy SolutionsHandango

Symbol TechnologiesIBMSMART TechnologiesVividLogicSynergy SolutionsHandango



Brad A. [email protected]


Brad A. [email protected]


Human Computer Interaction Institute Human Computer Interaction Institute

School of Computer ScienceSchool of Computer Science

Carnegie Mellon UniversityCarnegie Mellon University

Mobile Devices for Control of Ubiquitous Multimedia Brad A. Myers [email protected] Brad A. Myers...

Documents

Transcript of Mobile Devices for Control of Ubiquitous Multimedia Brad A. Myers [email protected] Brad A. Myers...