Recognizing Daily Routines Through Activity Spotting Ulf Blanke and Bernt Schiele Computer Science...

Recognizing Daily Routines Through Activity SpottingUlf Blanke and Bernt Schiele Computer Science Department, TU Darmstadt

Activity Recognition

Human Activity Recognition support awareness of applications (human to application)

support analysis of human activity (human to human)

Application Areas Healthcare: long term monitoring of patients (months!)

Elderly care: personal diary (weeks to months)

Industrial: workshop activities (minutes to a hours)

Recognizing Daily Routines Through Activity Spotting | Ulf Blanke and Bernt Schiele | 2

Towards High Level Activities

Low Level Activities E.g., walking, standing, biking… Lasting from seconds to minutes Detected by pose or characteristic motion.

High Level Activities E.g., daily routines: morning routine, dinner, working… More important for many domains (e.g., healthcare) Lasting from minutes to hours Consist of multiple low level activities:

approaching car(walking)

driving leaving car(walking)

commuting

preparing food eating doing dishes

dinner

… … … …


Related WorkMultilayer Approaches

commuting, working, lunch, dinner…

High-levelactivities

Low levelactivities

Sensor data

Zhang, D., Gatica-Perez, D., Bengio, S., McCowan, I., Lathoud, G. (CVPR 2004)Clarkson, B., Pentland, A. (ICASSP 1999)Oliver, N., Horvitz, E., Garg, A.: (Multimodal Interfaces 2002)

Mid-levelactivities

Morning Routine

wash

undressing drying

Cleaning teeth

scrub

e.g. walking, running, standing, sitting

shower Putting toothpaste

drying

. . . . . .

- Many parameters- Computationally intensive


Related WorkDirect Approach

commuting, working, lunch, dinner…

High-levelactivities

Low levelactivities

Sensor data

Mid-levelactivities

e.g. walking, running, standing, sitting

Huynh, T., Blanke, U., Schiele, B. (LoCA 2007)

Morning Routine

- High level activities exhibit high inner-class variability- All Data has to be considered


Can we learn distinctive parts of high level activities?

Can we reduce the amount of data important for recognition?

Research QuestionsActivity Spotting



LunchHigh-level activities

Low level activities

Sensor data

Which low level parts are important for high level activities?

Automatic selection

Dinner

walking picking up food

eatingPrepfood

eating Doing dishes



LunchHigh-level activities


Sensor data

Recognizing high level activities by activity spotting feasible?

Dinner

walking picking up food

eatingPrepfood

eating Doing dishes

Activity Spotting


Method

High-level activities


Sensor data

Doing dishes

K-means clusters

Joint boosting

Feature-Calculation


Low Level Activity Selection (Joint)Boosting

(1) Combination of low level activities to infer high-level activities

(2) Automatic Selection of most discriminative low level activities

(3) Sharing features (i.e. low level activities)

across high level activities

+

others


Experiment


Experimental SetupEvaluation Metrics

Quantitative Analysis

Tradeoff between precision & recall and

number of low-level selected activities?

how much data is needed (occurrence of activities used)?

Qualitative Analysis

Which activities are used – do they make sense?


7 days of a life from a single person[Huynh, T. - Ubicomp ‘08]

Two layers of annotation 4 high level routines, more than 20 low level routines

Experimental SetupDataset

Pocket

Wrist

Commuting Commuting

Working Working Dinner

Lunch

2 acceleration sensors

walking

standingin line

having a coffee

Lunch

walkingeating




Sensor data

Doing dishes

Feature-Calculation

Experimental SetupFixed Parameters

K-means clusters

Joint boosting

Mean and Variance - over 0.4s window- on (x,y,z)-acceleration- of pocket and wrist

Histograms- over 30min

window




Sensor data

Doing dishes

Feature-Calculation

Experimental SetupVaried Parameters

K-means clusters

Joint boosting

Jointboosting- rounds- Routines’ annotation

Kmean centers- soft and hard- K = 60


Quantitative Results

Soft Assignments

Rounds 4 10 80 Huynh 08

Precision in % 82.85 82.98 87.81 76.90

Recall in % 83.67 88.12 90.17 65.80

lowlevel activities in%

5.20 11.39 57.93 -

how much data? In % 12.78 17.74 74.30 -

4 10 80

72.71 77.34 86.40

82.67 82.39 90.32

5.19 12.14 50.82

2.11 4.94 45.42

Hard Assignments

Number of lowlevelactivities (clusters)

How much data

Rounds Rounds


Tradeoff

ResultsClassification scores for one day

Reducing number Low level activities

Precision lossat borders

Sco

res

80 rounds

10 rounds

4 rounds


Dinner Commute Lunch

sitting/desk activities (47.24%)

driving car (32.90%),

driving car (21.71%)

Time

walking (99.23%)

sitting / desk activities (97.86%)

walking (96.09%)

driving bike (47.86%) walking (22.51%) picking up food (16.81%)

queuing in line (43.86%) picking up food (14.59%)

driving bike (16.76%)

sitting/desk activities (31.20%)

36

6

42

48

29

13

53Lunch WorkCommuteDinner

Time

Time

Distribution of low level label for each selectedlow level cluster

Qualitative AnalysisWhich activities are used?


Dinner Commute Lunch

sitting/desk activities

driving car

driving car

Time

walking

sitting / desk activities

walking

driving bike walking picking up food

queuing in line picking up food

driving bike

sitting/desk activities

36

6

42

48

29

13

53Lunch WorkCommuteDinner

Time

Time

Distribution of low level label for each selectedlow level cluster

Qualitative AnalysisWhich activities are used?


Summary & Conclusion

Can we learn distinctive parts of high level activities? Yes.

Automatic Selection of important data

Top down perspective Find discriminative parts of a high level routines

Can we reduce the amount of data used for recognition? Yes.

Fraction (~5-8%) of data sufficient to recognize daily routines (~80%) Filter insignificant data reduce memory usage and computational costs suited for embedded long term activity recognition

Activity Spotting feasible for routine recognition.

Outperforms previous generative approaches on this dataset [Huynh - Ubicomp 08]


Thank you for your attention.Questions?

ご清聴、ありがとうございました。ご質問はありますか。

Dataset available at www.mis.tu-darmstadt.de


Date 08.05.2009 | Department of Computer Science | Ulf Blanke | 22

End of Presentation

(Joint)BoostingWeak Classifier

b: confidence, that sample is not part of classa: confidence, that sample is part of class

weak classifier

a b

Total confidence of separation

Strong classifier


Recognizing Daily Routines Through Activity Spotting Ulf Blanke and Bernt Schiele Computer Science...

Documents

Transcript of Recognizing Daily Routines Through Activity Spotting Ulf Blanke and Bernt Schiele Computer Science...