Tools for Activity Recognition: A portable kit of Sensors, the...
Transcript of Tools for Activity Recognition: A portable kit of Sensors, the...
Tools for Activity Recognition:A portable kit of Sensors, the Algorithms
and the Placelab
Emmanuel Munguia TapiaMIT Changing Places/House_n
Massachusetts Institute of Technology
Agenda
• Motivation: Health care crisis• Previous Work in Activity Recognition• The portable toolkit• Current Work at Intel• The MIT PlaceLab
Health Care Crisis • Possible solutions
– Make improvements existing health care system– Develop proactive home based approach to
keep people out of existing health care system
Health Care Crisis
Medical staff
Changes in behavior
(dementia/independence?)
Encouraging healthy behavior
Activity Recognition systems • Easy to retrofit in existing homes (HW)• Easy to adapt to user patterns (SW) • Address privacy concerns• Relatively inexpensive
Approaches– Manual– Automated
• Environment (invisible man)
• Body based
Sensors– High level– Low level
(binary)
Create Algorithms
• Data collected in natural settings• Reliable sensors• Ways to label activities (automated)
• ESM• Event triggered• Audio annotation• Speech recognition
• “Ground truth” to measure performance (Video?)• Guidelines on how to label activities
Sensors in the Environment
System overviewActivity labelsSensor firings
Classifier training
Activity likelihoods
Sensors in the Environment
Sensor installation locations
• 77 in home 1• 84 in home 2
Sensor Data Example
Zoom in
Multiclass naïve Bayesian classifier
Feature extraction
• Sensor fired• Temporal information
– Before, after– Duration Sensor 68 → Sensor 50
Feature extraction
• Sensor fired• Temporal information
– Before, after– Duration Sensor at drawer → Sensor at fridge
Feature extraction
• Sensor fired• Temporal information
– Before, after– Duration Sensor in kitchen → Sensor in Bathroom
Results Multiclass classifier (Subject 1)
Take away message
• Contribution– Ubiquitous but simple sensors may permit
automatic activity recognition – System deployed in multiple homes– Goal: recognition of ADLs – Results:
• Accuracies ranging from 25%-89%• Preliminary, but promising
• “Hoarder board” – Designed by V. Gerasimov– Saves data to CompactFlash card– 2-axis accelerometer data (ADXL2120E)– Axes in plane of the board– +- 10G– 76 Hz
Wearable Accelerometer Sensors
Target: 20 common domestic activities
• Walking• Sitting & relaxing• Standing still• Watching TV• Running• Stretching• Scrubbing• Folding laundry• Brushing teeth• Riding elevator
• Walking carrying items• Working on computer• Eating or drinking• Reading• Bicycling• Strength training• Vacuuming• Lying down & relaxing• Climbing stairs• Riding escalator
Accelerometer placement
• No wires to restrict motion• Hoarders (in soft case) attached
with elastic gauze
• 5 points–Right hip–Dominant wrist–Non-dominant upper arm–Dominant ankle–Non-dominant thigh
DT: leave-one-subject out
• Walking 89.7• Sitting & relaxing 94.8• Standing still 95.7• Watching TV 77.3• Running 87.7• Stretching 41.4• Scrubbing 81.1• Folding laundry 95.1• Brushing teeth 85.3• Riding elevator 43.6
• Walking carrying 82.1• Work computer 97.5• Eating/drink 88.7• Reading 91.8• Bicycling 96.3• Strength train 82.5• Vacuuming
96.4• Lying down 95.0• Climbing stairs 85.6• Riding escalator 70.6
Take Away• The study
–Activity recognition –5 non-wired accelerometers–Subjects labeled own data, often outside lab setting–20 household activities
• Results –Good results with decision tree classifier, even
when subjects collected own training data without researcher involvement
–Subject-specific training data for some activities may not be required
–Good performance can be achieved with only 2 of 5 accelerometers
My current researchat Intel
ADLs Activity Recognition Combining
• iGlove and RFID tags in objects(objects usage)
• Onbody acceleration (MITes sensors)
• Audio (3 sociometers)
Higher level activities from physical movements
If we can already recognize physical activities, then, we could
Eating meal• Walking, sitting down, eating, eating, drinking,
drinking, standing up, walking
Cleaning Bedroom• Walking, picking up smt, cleaning surface,
walking, sweeping, walking, ……
Algorithms
• HMMs Mixture of gaussian outputs• 2-3 Layer HMMs• Hierarchical HMMs• Dynamic Bayesian Networks
Back to MIT
Portable toolkit for collecting data in natural settings about activities
MITes: MIT Environmental sensors(Emmanuel Munguia Tapia and Natalia Marmasse)
State-Change MITesMeasure people’s interaction with objects in the
environment
• Stick-on and forget devices• Measure 2-axis acceleration(movement, tilt, and vibration)
• Relatively Inexpensive ~$30• Wireless 2.4GHz• Tx/Rx range ~30m indoor, 200m outdoors• Battery life ~1 year (worse case)• Active RFID tags that sense movement
State-Change MITes Installation
MITes Installation
• Objects where RFID tags are difficult to read
Wearable MITes• Add a daughter side board to measure 3-axis acceleration
Experience Sampling
Collect data from • Camera• Microphone• Hearth rate monitor• GPS• Accelerometers
(current work)
ESM is a robust data collection tool for acquiring self-reported data from subjects in experiments
Sampling • fixed• Statistical• Self-report
What has been the main problem faced in developing
the algorithms?
Create Algorithms
• Data collected in natural settings• Reliable sensors• Ways to label activities (automated)
• ESM• Event triggered• Audio annotation• Speech recognition
• “Ground truth” to measure performance (Video?)• Guidelines on how to label activities
Reinventing the wheel?• The AwareHome (Georgia Tech)• MavHome (U. Texas)• Microsoft home• The adaptive house (U.Colorado)• Smart Spaces Lab (NIST)• Intelligent room MIT
Demonstrations of projectsWorking space for testing projects/ideasResearch staff as subjectsReal subjects for couple of hours
PlaceLab • Is not a home of the future• Is not to show off “new technology”• Focus on research not demonstrations
Goals– Run different research studies– Real people living at PlaceLab 24/7 (weeks/months)– Collect necessary data for doing research
Design constraints– Reliable sensing infrastructure– Add/remove sensors on the fly (Modular)
Possible uses–Understanding complex human activity
–Encouraging healthy behavior/energy conservation
–Novel sensors and activity recognition
–Communication and Interface
–Indoor air quality control
–Biometric monitoring and every-day activity
–Lighting HVAC,appliance, entertainment, etc. control
–Privacy and trust (COUHES approved)
PlaceLab
PlaceLa Photos
Sensing infrastructureReliable sensing• Wired in fixed objects (1-wire)• Wireless in movable objects (MITes)
• Sensors (and other)– Color and IR video– Reed switches– Environmental– Flow– Current– Microphones– IR/RF position tracking– Digital light/speakers– Acceleration/biometric
– Temperature– Humidity– B. Pressure– CO, CO2– Light intensity
Prior approaches to
(Intille et al., 2003)
Prototype cabinet
Switch State Sensors
Water Flow Sensors
State-Change MITes
Audio Input
Current Sensors
Environmental Sensors
Thank you!• For more information on the House_n
PlaceLab,and portable sensor toolkit, contact
– Emmanuel Munguia [email protected]
– Stephen [email protected]
– Kent [email protected]