Tools for Activity Recognition:A portable kit of Sensors, the Algorithms

and the Placelab

Emmanuel Munguia TapiaMIT Changing Places/House_n

Massachusetts Institute of Technology

Agenda

• Motivation: Health care crisis• Previous Work in Activity Recognition• The portable toolkit• Current Work at Intel• The MIT PlaceLab

Health Care Crisis • Possible solutions

– Make improvements existing health care system– Develop proactive home based approach to

keep people out of existing health care system

Health Care Crisis

Medical staff

Changes in behavior

(dementia/independence?)

Encouraging healthy behavior

Activity Recognition systems • Easy to retrofit in existing homes (HW)• Easy to adapt to user patterns (SW) • Address privacy concerns• Relatively inexpensive

Approaches– Manual– Automated

• Environment (invisible man)

• Body based

Sensors– High level– Low level

(binary)

Create Algorithms

• Data collected in natural settings• Reliable sensors• Ways to label activities (automated)

• ESM• Event triggered• Audio annotation• Speech recognition

• “Ground truth” to measure performance (Video?)• Guidelines on how to label activities

Sensors in the Environment

System overviewActivity labelsSensor firings

Classifier training

Activity likelihoods

Sensors in the Environment

Sensor installation locations

• 77 in home 1• 84 in home 2

Sensor Data Example

Zoom in

Multiclass naïve Bayesian classifier

Feature extraction

• Sensor fired• Temporal information

– Before, after– Duration Sensor 68 → Sensor 50

Feature extraction

• Sensor fired• Temporal information

– Before, after– Duration Sensor at drawer → Sensor at fridge

Feature extraction

• Sensor fired• Temporal information

– Before, after– Duration Sensor in kitchen → Sensor in Bathroom

Results Multiclass classifier (Subject 1)

Take away message

• Contribution– Ubiquitous but simple sensors may permit

automatic activity recognition – System deployed in multiple homes– Goal: recognition of ADLs – Results:

• Accuracies ranging from 25%-89%• Preliminary, but promising

• “Hoarder board” – Designed by V. Gerasimov– Saves data to CompactFlash card– 2-axis accelerometer data (ADXL2120E)– Axes in plane of the board– +- 10G– 76 Hz

Wearable Accelerometer Sensors

Target: 20 common domestic activities

• Walking• Sitting & relaxing• Standing still• Watching TV• Running• Stretching• Scrubbing• Folding laundry• Brushing teeth• Riding elevator

• Walking carrying items• Working on computer• Eating or drinking• Reading• Bicycling• Strength training• Vacuuming• Lying down & relaxing• Climbing stairs• Riding escalator

Accelerometer placement

• No wires to restrict motion• Hoarders (in soft case) attached

with elastic gauze

• 5 points–Right hip–Dominant wrist–Non-dominant upper arm–Dominant ankle–Non-dominant thigh

DT: leave-one-subject out

• Walking 89.7• Sitting & relaxing 94.8• Standing still 95.7• Watching TV 77.3• Running 87.7• Stretching 41.4• Scrubbing 81.1• Folding laundry 95.1• Brushing teeth 85.3• Riding elevator 43.6

• Walking carrying 82.1• Work computer 97.5• Eating/drink 88.7• Reading 91.8• Bicycling 96.3• Strength train 82.5• Vacuuming

96.4• Lying down 95.0• Climbing stairs 85.6• Riding escalator 70.6

Take Away• The study

–Activity recognition –5 non-wired accelerometers–Subjects labeled own data, often outside lab setting–20 household activities

• Results –Good results with decision tree classifier, even

when subjects collected own training data without researcher involvement

–Subject-specific training data for some activities may not be required

–Good performance can be achieved with only 2 of 5 accelerometers

My current researchat Intel

ADLs Activity Recognition Combining

• iGlove and RFID tags in objects(objects usage)

• Onbody acceleration (MITes sensors)

• Audio (3 sociometers)

Higher level activities from physical movements

If we can already recognize physical activities, then, we could

Eating meal• Walking, sitting down, eating, eating, drinking,

drinking, standing up, walking

Cleaning Bedroom• Walking, picking up smt, cleaning surface,

walking, sweeping, walking, ……

Algorithms

• HMMs Mixture of gaussian outputs• 2-3 Layer HMMs• Hierarchical HMMs• Dynamic Bayesian Networks

Back to MIT

Portable toolkit for collecting data in natural settings about activities

MITes: MIT Environmental sensors(Emmanuel Munguia Tapia and Natalia Marmasse)

State-Change MITesMeasure people’s interaction with objects in the

environment

• Stick-on and forget devices• Measure 2-axis acceleration(movement, tilt, and vibration)

• Relatively Inexpensive ~$30• Wireless 2.4GHz• Tx/Rx range ~30m indoor, 200m outdoors• Battery life ~1 year (worse case)• Active RFID tags that sense movement

State-Change MITes Installation

MITes Installation

• Objects where RFID tags are difficult to read

Wearable MITes• Add a daughter side board to measure 3-axis acceleration

Experience Sampling

Collect data from • Camera• Microphone• Hearth rate monitor• GPS• Accelerometers

(current work)

ESM is a robust data collection tool for acquiring self-reported data from subjects in experiments

Sampling • fixed• Statistical• Self-report

What has been the main problem faced in developing

the algorithms?

Create Algorithms

• Data collected in natural settings• Reliable sensors• Ways to label activities (automated)

• ESM• Event triggered• Audio annotation• Speech recognition

• “Ground truth” to measure performance (Video?)• Guidelines on how to label activities

Reinventing the wheel?• The AwareHome (Georgia Tech)• MavHome (U. Texas)• Microsoft home• The adaptive house (U.Colorado)• Smart Spaces Lab (NIST)• Intelligent room MIT

Demonstrations of projectsWorking space for testing projects/ideasResearch staff as subjectsReal subjects for couple of hours

PlaceLab • Is not a home of the future• Is not to show off “new technology”• Focus on research not demonstrations

Goals– Run different research studies– Real people living at PlaceLab 24/7 (weeks/months)– Collect necessary data for doing research

Design constraints– Reliable sensing infrastructure– Add/remove sensors on the fly (Modular)

Possible uses–Understanding complex human activity

–Encouraging healthy behavior/energy conservation

–Novel sensors and activity recognition

–Communication and Interface

–Indoor air quality control

–Biometric monitoring and every-day activity

–Lighting HVAC,appliance, entertainment, etc. control

–Privacy and trust (COUHES approved)

PlaceLab

PlaceLa Photos

Sensing infrastructureReliable sensing• Wired in fixed objects (1-wire)• Wireless in movable objects (MITes)

• Sensors (and other)– Color and IR video– Reed switches– Environmental– Flow– Current– Microphones– IR/RF position tracking– Digital light/speakers– Acceleration/biometric

– Temperature– Humidity– B. Pressure– CO, CO2– Light intensity

Prior approaches to

(Intille et al., 2003)

Prototype cabinet

Switch State Sensors

Water Flow Sensors

State-Change MITes

Audio Input

Current Sensors

Environmental Sensors

Thank you!• For more information on the House_n

PlaceLab,and portable sensor toolkit, contact

– Emmanuel Munguia Tapiaemunguia@media.mit.edu

– Stephen Intilleintille@mit.edu

– Kent LarsonKll@mit.edu