Tools for Activity Recognition: A portable kit of Sensors, the...

59
Tools for Activity Recognition: A portable kit of Sensors, the Algorithms and the Placelab Emmanuel Munguia Tapia MIT Changing Places/House_n Massachusetts Institute of Technology

Transcript of Tools for Activity Recognition: A portable kit of Sensors, the...

Page 1: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Tools for Activity Recognition:A portable kit of Sensors, the Algorithms

and the Placelab

Emmanuel Munguia TapiaMIT Changing Places/House_n

Massachusetts Institute of Technology

Page 2: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Agenda

• Motivation: Health care crisis• Previous Work in Activity Recognition• The portable toolkit• Current Work at Intel• The MIT PlaceLab

Page 3: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Health Care Crisis • Possible solutions

– Make improvements existing health care system– Develop proactive home based approach to

keep people out of existing health care system

Page 4: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Health Care Crisis

Medical staff

Changes in behavior

(dementia/independence?)

Encouraging healthy behavior

Page 5: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Activity Recognition systems • Easy to retrofit in existing homes (HW)• Easy to adapt to user patterns (SW) • Address privacy concerns• Relatively inexpensive

Approaches– Manual– Automated

• Environment (invisible man)

• Body based

Sensors– High level– Low level

(binary)

Page 6: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Create Algorithms

• Data collected in natural settings• Reliable sensors• Ways to label activities (automated)

• ESM• Event triggered• Audio annotation• Speech recognition

• “Ground truth” to measure performance (Video?)• Guidelines on how to label activities

Page 7: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Sensors in the Environment

Page 8: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

System overviewActivity labelsSensor firings

Classifier training

Activity likelihoods

Page 9: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Sensors in the Environment

Page 10: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 11: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Sensor installation locations

• 77 in home 1• 84 in home 2

Page 12: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Sensor Data Example

Page 13: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Zoom in

Page 14: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Multiclass naïve Bayesian classifier

Page 15: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Feature extraction

• Sensor fired• Temporal information

– Before, after– Duration Sensor 68 → Sensor 50

Page 16: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Feature extraction

• Sensor fired• Temporal information

– Before, after– Duration Sensor at drawer → Sensor at fridge

Page 17: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Feature extraction

• Sensor fired• Temporal information

– Before, after– Duration Sensor in kitchen → Sensor in Bathroom

Page 18: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Results Multiclass classifier (Subject 1)

Page 19: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Take away message

• Contribution– Ubiquitous but simple sensors may permit

automatic activity recognition – System deployed in multiple homes– Goal: recognition of ADLs – Results:

• Accuracies ranging from 25%-89%• Preliminary, but promising

Page 20: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

• “Hoarder board” – Designed by V. Gerasimov– Saves data to CompactFlash card– 2-axis accelerometer data (ADXL2120E)– Axes in plane of the board– +- 10G– 76 Hz

Wearable Accelerometer Sensors

Page 21: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Target: 20 common domestic activities

• Walking• Sitting & relaxing• Standing still• Watching TV• Running• Stretching• Scrubbing• Folding laundry• Brushing teeth• Riding elevator

• Walking carrying items• Working on computer• Eating or drinking• Reading• Bicycling• Strength training• Vacuuming• Lying down & relaxing• Climbing stairs• Riding escalator

Page 22: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Accelerometer placement

• No wires to restrict motion• Hoarders (in soft case) attached

with elastic gauze

• 5 points–Right hip–Dominant wrist–Non-dominant upper arm–Dominant ankle–Non-dominant thigh

Page 23: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 24: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

DT: leave-one-subject out

• Walking 89.7• Sitting & relaxing 94.8• Standing still 95.7• Watching TV 77.3• Running 87.7• Stretching 41.4• Scrubbing 81.1• Folding laundry 95.1• Brushing teeth 85.3• Riding elevator 43.6

• Walking carrying 82.1• Work computer 97.5• Eating/drink 88.7• Reading 91.8• Bicycling 96.3• Strength train 82.5• Vacuuming

96.4• Lying down 95.0• Climbing stairs 85.6• Riding escalator 70.6

Page 25: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Take Away• The study

–Activity recognition –5 non-wired accelerometers–Subjects labeled own data, often outside lab setting–20 household activities

• Results –Good results with decision tree classifier, even

when subjects collected own training data without researcher involvement

–Subject-specific training data for some activities may not be required

–Good performance can be achieved with only 2 of 5 accelerometers

Page 26: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

My current researchat Intel

Page 27: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

ADLs Activity Recognition Combining

• iGlove and RFID tags in objects(objects usage)

• Onbody acceleration (MITes sensors)

• Audio (3 sociometers)

Page 28: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Higher level activities from physical movements

If we can already recognize physical activities, then, we could

Eating meal• Walking, sitting down, eating, eating, drinking,

drinking, standing up, walking

Cleaning Bedroom• Walking, picking up smt, cleaning surface,

walking, sweeping, walking, ……

Page 29: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Algorithms

• HMMs Mixture of gaussian outputs• 2-3 Layer HMMs• Hierarchical HMMs• Dynamic Bayesian Networks

Page 30: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Back to MIT

Page 31: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Portable toolkit for collecting data in natural settings about activities

MITes: MIT Environmental sensors(Emmanuel Munguia Tapia and Natalia Marmasse)

Page 32: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

State-Change MITesMeasure people’s interaction with objects in the

environment

• Stick-on and forget devices• Measure 2-axis acceleration(movement, tilt, and vibration)

• Relatively Inexpensive ~$30• Wireless 2.4GHz• Tx/Rx range ~30m indoor, 200m outdoors• Battery life ~1 year (worse case)• Active RFID tags that sense movement

Page 33: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

State-Change MITes Installation

Page 34: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

MITes Installation

• Objects where RFID tags are difficult to read

Page 35: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Wearable MITes• Add a daughter side board to measure 3-axis acceleration

Page 36: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Experience Sampling

Collect data from • Camera• Microphone• Hearth rate monitor• GPS• Accelerometers

(current work)

ESM is a robust data collection tool for acquiring self-reported data from subjects in experiments

Sampling • fixed• Statistical• Self-report

Page 37: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

What has been the main problem faced in developing

the algorithms?

Page 38: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Create Algorithms

• Data collected in natural settings• Reliable sensors• Ways to label activities (automated)

• ESM• Event triggered• Audio annotation• Speech recognition

• “Ground truth” to measure performance (Video?)• Guidelines on how to label activities

Page 39: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 40: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Reinventing the wheel?• The AwareHome (Georgia Tech)• MavHome (U. Texas)• Microsoft home• The adaptive house (U.Colorado)• Smart Spaces Lab (NIST)• Intelligent room MIT

Demonstrations of projectsWorking space for testing projects/ideasResearch staff as subjectsReal subjects for couple of hours

Page 41: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

PlaceLab • Is not a home of the future• Is not to show off “new technology”• Focus on research not demonstrations

Goals– Run different research studies– Real people living at PlaceLab 24/7 (weeks/months)– Collect necessary data for doing research

Design constraints– Reliable sensing infrastructure– Add/remove sensors on the fly (Modular)

Page 42: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Possible uses–Understanding complex human activity

–Encouraging healthy behavior/energy conservation

–Novel sensors and activity recognition

–Communication and Interface

–Indoor air quality control

–Biometric monitoring and every-day activity

–Lighting HVAC,appliance, entertainment, etc. control

–Privacy and trust (COUHES approved)

Page 43: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 44: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

PlaceLab

Page 45: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

PlaceLa Photos

Page 46: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Sensing infrastructureReliable sensing• Wired in fixed objects (1-wire)• Wireless in movable objects (MITes)

• Sensors (and other)– Color and IR video– Reed switches– Environmental– Flow– Current– Microphones– IR/RF position tracking– Digital light/speakers– Acceleration/biometric

– Temperature– Humidity– B. Pressure– CO, CO2– Light intensity

Page 47: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Prior approaches to

(Intille et al., 2003)

Page 48: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Prototype cabinet

Page 49: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 50: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Switch State Sensors

Page 51: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Water Flow Sensors

Page 52: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

State-Change MITes

Page 53: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Audio Input

Page 54: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 55: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Current Sensors

Page 56: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Environmental Sensors

Page 57: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 58: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •
Page 59: Tools for Activity Recognition: A portable kit of Sensors, the ...alumni.media.mit.edu/~emunguia/pdf/MIT house_n.pdfSensors – High level –Low level (binary) Create Algorithms •

Thank you!• For more information on the House_n

PlaceLab,and portable sensor toolkit, contact

– Emmanuel Munguia [email protected]

– Stephen [email protected]

– Kent [email protected]