LINCS

Context Aware Computing Context Aware Computing

Dan Dan SiewiorekSiewiorek

Asim Smailagic Asim Smailagic

QUALCOMM Visit QUALCOMM Visit –– 5/20/085/20/08

LINCS

Vision

• Our vision is to create intelligent systems that augment human capabilities and mind

LINCS

Research Thrusts

• Machine Learning• Computational Requirements (Power)• Human in Loop

LINCS

Machine Learning

• Platforms

• Activity Recognition

• Location Recognition

• Wheel chair Propulsion and Location Recognition

• Psychosocial Stress Measurements

LINCS

Platform Functions

• Accelerometers used for activity and gesture recognition

• Light and audio sensor signatures used for location recognition

• Audio sensor used to detect if person is talking / engaged in conversation

• LEDs used for different visual notifications

LINCS

Feature Space of Activities

LINCS

Wheelchair Propulsion Pattern Recognition

Semicircular Single Looping over Propulsion

Double Looping over Propulsion

Arcing

LINCS

Results

• Based on 142 samples over four different surfaces: medium pile carpet, deep pile carpet, tile, parking lot

• Arcing accuracy low because it is often misclassified as semicircular

90.1%Average

74.1%Arcing

97.4%Double Looping

92.1%Semicircular

92.1%Single Looping

AccuracyPropulsion Pattern

Location Study for WheelchairLight, audio and acceleration data is collectedThe location is annotated using an application on the eWatch on wristThe eWatches are synchronized before data collection begins

eWatch on Wrist eWatch on Frame

LINCS

Real-Time Assessment of Psychosocial Stress

• With eWatch for data collection, even assembly workers, bus drivers can handle interruptions of their daily routines to answer questions about their stress

• Different types of prompting, input (question) and output (answer)

• Evaluation in respect to: easy of use, subject compliance and data quality, utility across a wide spectrum of populations, and training demands

LINCS

Modalities of Interaction

• Input Modalities – Buttons– Voice (Speech)– Tap– Gestures

• Output Modalities– Vibration– LCD Screen– LEDs– Beep

LINCS

Power

- Sampling Rate- Selective Sampling- Human Behavior

LINCS

Sampling Rate

Classification Accuracy vsSampling Rate

Classification Accuracy vsEstimated Battery Lifetime

LINCS

Do we really need every sample?

• There is a significant amount of inertia to human behaviors– Periodic sampling can remove at least a portion of

the redundant samples• Human behaviors have different duration

profiles– Can modify sampling strategy given current user

context to improve balance of samples• User study, 100 hours of beeper study data

LINCS

Performance Metrics and Results

• Accuracy– Percentage of time that the system predicted activity matches the true,

current activity– Using only 10% of the available sample windows, Markov model achieves

98% accuracy

• Missed Activities– Percentage of activities during which no samples are taken– Using 10% of available sample windows, Markov model missed 5% of

activities

• Average Latency– Average number of seconds between the true transition between two

activities and detection of the transition– Using 10% of available sample windows, Markov model latency is 15

seconds

LINCS

Human in Loop

• Power and User Efficient Interactions with Mobile Systems– Predicting User and Energy Performance During

Early Design – 5%-9% User Time Prediction Error,

4%-8% Task Energy Prediction Error• RADAR: Personal Assistant That Learns• Virtual Coach

LINCS

KLEM: Keystroke Level Energy Modeling

• Extending KLM to predict task energyGiven: a task, the methods to execute the task, the design, and a target platformPredict: user time and system energy consumption of the task

LINCS

Comparing Design AlternativesComparison: different modalities

0.002.004.006.008.00

10.0012.0014.00

Speech no LCDList D

ragMap

Nav

List Tap

and H

oldList T

rough

List Key

List Arro

wSpeec

hList H

W Butto

n

Energy (joule)

Time (sec)

Comparison: different input methods

0.002.004.006.008.00

10.0012.0014.0016.0018.00

Speech no LCD

Speech

Transc

ript R

ecognitio

n

Soft Keyb

oardLette

r Rec

ognition

Energy (joule)

Time (sec)

List Trough List Arrow

List Tap&Hold List Drag

List HW Button

Same designDifferent modes

LINCS

RADAR: Personal Assistant that LearnsConference Planning Task

Just before a major 4-day conference, the schedule is disrupted, e.g., by weather, loss of venue, strike, travel disruption…

The user (and RADAR) must assess the situation, develop a revised plan, get the word out about the new plan, and deal with queries and requests during the crisis.

The results are evaluated on:• Quality and completeness of the new plan• Successful completion of related tasks• Costs of the solutions

Website

Crisis Manager

Conference Participants

Conference Organizers

?

ConferenceSchedule RADAR

General Planning,Priority Setting,

Meta-level reasoning

Email & NaturalLanguage

ProcessingE-mail

Radar Console (User

Interface)User

Task Manager Calendar Manager

Webmaster

Briefings

Space/Time Planner

E-Mail Manager

Specialist Modules

Task/InfoBus

InferenceEngine

SharedKnowledge Base

Episodic Memory

Arch & KBArch & KBEmail Mgt Email Mgt PlanningPlanningBriefingBriefing

General Planning,Priority Setting,

Meta-level reasoning

Email & NaturalLanguage

ProcessingE-mail

Radar Console (User

Interface)User

Task Manager Calendar Manager

Webmaster

Briefings

Space/Time Planner

E-Mail Manager

Specialist Modules

Task/InfoBus

InferenceEngine

SharedKnowledge Base

Episodic Memory

InferenceEngine

SharedKnowledge Base

Episodic Memory

Arch & KBArch & KBEmail Mgt Email Mgt PlanningPlanningBriefingBriefing

Arch & KBArch & KBEmail Mgt Email Mgt PlanningPlanningBriefingBriefing

RevisedSchedule

LINCS

Double Log PlotConstraintEntry

“Start Briefing”Pop-Up

User SwitchesTo VOT

Console BriefingTask Viewed

User StartedBA

User OptimizesAnd PublishesSchedule

Well-PerformingUser

LostUser

User incorrectly starts session by doing an STP optimize

User switches to VOT as suggested after publish

User realizes mistake and begins entering constrains

After break, user switches among many different tasks

User publishes schedule again

User switches toVOT again after publish

User starts the BA too early

LINCS

Subject Interruption Annoyance Ratings (1 Low,10 High)

LINCS

Attention Manager: Improving User Productivity

• Monitor user inputs and task activity• Provide reminders to users to perform important

functions as deadlines approach• Pick appropriate time to interact with user to

avoid context swapping and user thrashing

LINCS

Virtual Coach - Overview

1m /s FSR

0

5

10

15

20

25

30

5000 5500 6000 6500 7000 7500 8000 8500 9000 9500 10000

Tim e (m illisecond)

Forc

e (N

/cm

2 )

Anaerobic Exercise

Aerobic Exercise

Seated Activity

Wheel Chair Propulsion

Mobile Sensors

LINCS

Virtual Coach Architecture

Coaching DomainModel

Personal AgencyAdjust capability of Interaction

Remind-ing

Labeled Data Base

Perception & AwarenessSensing

Auth-oring

User InteractionInput

User Engagement

OutputPrescription

LINCS

Virtual Coach

PRESSURE SENSORS

IR

IR

Tilt

LINCS

Examples of Related Applications

- Locator- Aircraft Maintenance- Context Aware Car Manuals - GM

LINCS

Locator@CMU System Diagram

LINCS

Mobility Percentages

0%5%

10%15%20%25%30%35%40%45%50%

1 2 3 4 5 6 7 8 9 10+

Mobility (Access Points per Day)

Perc

ent o

f Use

rs

2003 2005

0%5%

10%15%20%25%30%35%40%45%50%

1 2 3 4 5 6 7 8 9 10+

Mobility (Access Points per Day)

Perc

ent o

f Use

rs

2003 2005

LINCS

Heavy Equipment

• Finishing up his recording of new annotation, he clicks the select buttonto stop recording

• The new annotation is displayed under the search results

LINCS

User Interaction: GM Polaris: Context Aware Owner’s Manual

TouchPad(EdgeWrite)

TouchScreen LCD

PowerSwitch

AWARE

LINCS

Future Research• More extensive field studies• Further improvements with eWatch platform

– Newly upgraded, new housing– Complete redesign

• Further advances in power saving methods• Ruggedized Power Wheel Chair Virtual Coach• Further developed ADEPT Instrumentation and

Language Technology for in-process monitoring• Determine user intent from primary sensors of physical

activity, location, biological signs• Make systems responsible for their own management

and evolution• Synthesis of new functionality for virtual coach to adapt

to changes