Robust Expert Systems for more Flexible Real-World

Activity Recognition

Granada, Friday, April 25, 2014

Presented by: Oresti Baños

Supervised by: Miguel Damas, Héctor Pomares and Ignacio Rojas Department of Computer Architecture and Computer Technology,

CITIC-UGR, University of Granada, SPAIN

Human Activity

2

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Health

Abnormal behavior detection

Proactive Assistance

Labour risk prevention

Wellness

Sports

Gaming

Human Activity

• Why is identifying human activity interesting?

3

REPORT (AWARENESS, MOTIVATIONAL,…) - USER - THIRD PARTIES ACT ON OUR BEHALF (TRIGGER ALERTS, TURN THE LIGHTS ON,…)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Activity Recognition (AR)

• Activity recognition concept

“Recognize the actions and goals of one or more agents from a series of observations on the agents' actions and the environmental conditions”

• Activity recognition process

4

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Ambient camera y micro pero predominan los wearables: se pueden usar outdoor, directamente en el cuerpo, privacidad… Hay varias formas, fuera indoor

Phenomena

Human activity (body motion)

Measurement

Sensing (ambient/wearables)

Processing

Data adequation and knowledge

inference

Recognized Activity

Wearable Activity Recognition

5

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

The major limitation within the use of wearable computing has been obtrusiveness… Technology miniaturization has helped to surpass this issue as we are withnessing during the last months. Each day we see a new gadget which further promises full recognition capabilities… but is it true?

• Wearable activity recognition systems are ready!

The first system capable of fully

recognize your daily routine.

AtlasWearables (2014)

The simplest way to understand

your day and night.

Jawbone Up (2014)

The best activity tracker on the

market. Fitbit Force (2014)

The device that tracks your active

life and measures all kind of

activities. Nike Fuel (2014)

Wearable Activity Recognition

6

Is wearable computing mature enough? News: - http://mobihealthnews.com/31697/sur

vey-one-third-of-wearable-device-owners-stopped-using-them-within-six-months/

- http://nothingtosay.com/2013/11/wearable-gadgets-of-various-and-dubious-value/

- http://www.technologyreview.com/news/521931/fitness-trackers-still-need-to-work-out-kinks/

- http://www.elmundo.es/blogs/elmundo/el-gadgetoblog/2014/03/12/pulseras-inteligentes-no-tanto.html

- http://well.blogs.nytimes.com/2014/03/10/activity-trackers-dont-sense-everything/?_php=true&_type=blogs&smid=tw-nytimes&_r=0

- http://www.crowdfundinsider.com/2014/04/35002-indiegogos-reputation-may-hinge-outcome-gobe-campaign/

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

The major limitation within the use of wearable computing has been obtrusiveness… Technology miniaturization has helped to surpass this issue as we are withnessing during the last months. Each day we see a new gadget which further promises full recognition capabilities… but is it true?

• But… do wearable activity recognition systems meet people’s expectance?

SO YOU’RE TELLING ME

A BRACELET CAN TRACK MY COMPLETE ACTIVITY

LONGLIFE?

Challenges for Real-World Activity Recognition

• Actively investigated:

– Reliability

– Simplicity

– Latency

• Barely addressed:

– Privacy

– Fault-tolerance

– Usability

– Unobtrusiveness

– Fashionability

– Self-configuration

– Auto-adaptation

– Evolvability

7

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Challenges for Real-World Activity Recognition

8

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Buscar imagen de fault-tolerance

• Actively investigated:

– Reliability

– Simplicity

– Latency

• Barely addressed:

– Privacy

– Fault-tolerance

– Usability

– Unobtrusiveness

– Fashionability

– Self-configuration

– Auto-adaptation

– Evolvability

Thesis Motivation and Objectives

• Motivation:

“Create more advanced systems capable of handling real-world AR issues as well as to incorporate more intelligent capabilities to transform experimental prototypes into actual usable applications”

• Objectives:

– O1: “Investigate the tolerance of standard AR systems to unforeseen sensor failures and faults, as well as contribute with an alternate approach to cope with these technological anomalies” Fault-tolerance

– O2: “Research the robustness of standard AR systems to unforeseen variations in the sensor deployment, as well as contribute with an alternate approach to cope with these practical anomalies” Usability, Unobtrusiveness

– O3: “Study the capacity of standard AR systems to support unforeseen changes in the sensor network, as well as contribute with an alternate approach to cope with these topological variations” Self-configuration, auto-adaptation, evolvability

9

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Activity Recognition Process

10

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Phenomena

Human activity (body motion)

Measurement

Sensing (ambient/wearables)

Processing

Data adequation and knowledge

inference

Recognized Activity

How does it work exactly?

Activity Recognition Process

11

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

The Activity Recognition Chain (ARC)

Phenomena

Human activity (body motion)

Measurement

Sensing (ambient/wearables)

Processing

Data adequation and knowledge

inference

Recognized Activity

Activity Recognition Chain (ARC)

12

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Activity Recognition Chain (ARC)

13

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

During the acquisition maybe explain accelerometers and how they work (video)

Activity Recognition Chain (ARC)

14

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Activity Recognition Chain (ARC)

15

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Colador y pan, OUT! Cambiar por otro…

Activity Recognition Chain (ARC)

16

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Activity Recognition Chain (ARC)

17

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Activity Recognition Chain (ARC)

18

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Tolerance of AR systems to sensor faults and failures

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Objective: “Investigate the tolerance of standard AR systems to unforeseen sensor failures and faults, as well as contribute with an alternate approach

to cope with these technological anomalies”

Problem Statement

20

SENSOR ERRORS

Are standard activity recognition systems prepared to cope with sensor technological anomalies?

Is it possible to keep the systems functioning under the effects of sensor errors?

Activity recognition process

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Body motion sensing

Signal processing and

reasoning

Recognition of activities

Signal effects

Sensor Technological Anomalies

21

• Faults (overheating, environmental changes, decalibration)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

• Failures (outages, breakdowns, disconnection, battery depletion)

Necesidad de SotA?

Sensor Technological Anomalies in AR: Related Work

• Detection of sensor anomalies

– Sensor query (Rost06)

– Neighborhood data correlation

• Signal level (Yao10)

• Feature level (Ramanathan09)

• Reasoning (Rajasegarar07, Ganeriwal08)

• Counteraction of sensor anomalies

– Data imputation (Uchida13)

– Sensor fusion (Sagha13)

S. Rost and H. Balakrishnan. Memento: A health monitoring system for wireless sensor networks. In 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, volume 2, pp. 575-584, 2006. Y. Yao, A. Sharma, L. Golubchik, and R. Govindan. Online anomaly detection for sensor systems: A simple and efficient approach. Performance Evaluation, 67(11):1059-1075, November 2010. N. Ramanathan, T. Schoellhammer, E. Kohler, K. Whitehouse, T. Harmon, and D. Estrin. Suelo: human-assisted sensing for exploratory soil monitoring studies. In Proceedings of the 7th ACM Conference on Embedded Networked Sensor Systems, pp. 197-210, 2009. S. Rajasegarar, C. Leckie, M. Palaniswami, and J. C. Bezdek. Quarter sphere based distributed anomaly detection in wireless sensor networks. In IEEE International Conference on Communications, pp. 3864-3869, June 2007. S. Ganeriwal, L. K. Balzano, and M. B. Srivastava. Reputationbased framework for high integrity sensor networks. ACM Transaction on Sensor Networks, 4(3):1-37, June 2008. R. Uchida, H. Horino, and R. Ohmura. Improving fault tolerance of wearable wearable sensor-based activity recognition techniques. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 633-644, 2013. H. Sagha, H. Bayati, J. del R. Millan, and R. Chavarriaga. On-line anomaly detection and resilience in classifier ensembles. Pattern Recognition Letters, 34(15):1916-1927, 2013

22

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor Failures in Classic AR Systems

• Single-sensor ARC (SARC)

23

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

• Single-sensor ARC (SARC)

24

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

If the sensor fails, the complete system fails

Solution Use more sensors for redundancy

(multi-sensor ARC or MARC)

Sensor Failures in Standard AR Systems

• Feature fusion multi-sensor ARC (FFMARC)

25

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor Failures in Standard AR Systems

• Feature fusion multi-sensor ARC (FFMARC)

26

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

If a sensor fails, the complete system fails

Solution Independent ARCs + decision fusion

Sensor Failures in Standard AR Systems

• Decision fusion multi-sensor ARC (DFMARC)

27

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor Failures in Standard AR Systems

• Decision fusion multi-sensor ARC (DFMARC)

28

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

If a sensor fails, the system is still capable of functioning

but…

Is it capable of recognition?

Sensor Failures in Standard AR Systems

• Hierarchical decision (HD)

– Information from some sensors more valuable than from others (e.g., body part for a certain activity) Ranking of decisions

– Decisions mainly made on top (recognition relies on a sensor or few sensors) Problem when top-ranked sensors get unavailable

• Majority voting (MV)

– Equality scheme (all sensors have the same importance) Fairness, decisiveness

– A plurality of weak decisors may prevail over the rest Tyranny of the majority

29

Sensor Failures in Standard AR Systems

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

…

c11,c12,…,c1k

c1=φ(c11,c21,…, cM1), c2=φ(c12,c22,…, cM2),

… ck=φ(c1k,c2k,…, cMk)

c21,c22,…, c2k

cM1,cM2,…,cMk

…

…

…

…

…

…

c11,c12,…,c1k

c1=φ(c11,c21,…, cM1), c2=φ(c12,c22,…, cM2),

… ck=φ(c1k,c2k,…, cMk)

c21,c22,…, c2k

cM1,cM2,…,cMk

…

A Novel Method: Hierarchical Weighted Classifier

30

Sensor M

Sensor 2

SM

S2

S1 α11

Ψ

C12

C1N

C11

Ψ

C21

C22

C2N

Ψ

CM1

CM2

CMN

Ψ

Decisio

n

Activity level

(base classifier)

Sensor level

(sensor classifier)

Network level

(sensor fusion)

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

Sensor 1

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

31

Sensor M

Sensor 2

SM

S2

S1 α11

Ψ

C12

C1N

C11

Ψ

C21

C22

C2N

Ψ

CM1

CM2

CMN

Ψ

Decisio

n

Activity level

(base classifier)

Sensor level

(sensor classifier)

Network level

(sensor fusion)

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

Sensor 1

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

A Novel Method: Hierarchical Weighted Classifier N activities & M sensors

32

Sensor M

Sensor 2

SM

S2

S1 α11

Ψ

C12

C1N

C11

Ψ

C21

C22

C2N

Ψ

CM1

CM2

CMN

Ψ

Decisio

n

Activity level

(base classifier)

Sensor level

(sensor classifier)

Network level

(sensor fusion)

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

Sensor 1

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

A Novel Method: Hierarchical Weighted Classifier N activities & M sensors

S1 α11

Ψ

C12

C1N

C11 β11

α12 β12

α1N β1N

𝑞𝑚 𝑥𝑚𝑘= 𝑎𝑟𝑔𝑚𝑎𝑥

𝑞

𝑂𝑚 𝑥𝑚𝑘

𝑂𝑚 𝑥𝑚𝑘= 𝑊𝐷𝑚𝑛 𝑥𝑚𝑘

𝑁

𝑛=1

𝛼𝑚𝑛 =𝑇𝑃𝑚𝑛

𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛

𝛽𝑚𝑛 =𝑇𝑁𝑚𝑛

𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛

𝑊𝐷𝑚𝑛 𝑥𝑚𝑘=

𝛼𝑚𝑛, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

0, 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

∀𝑞 = 𝑛

𝛽𝑚𝑛 , 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

0, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

∀𝑞 ≠ 𝑛

33

Sensor M

Sensor 2

SM

S2

S1 α11

Ψ

C12

C1N

C11

Ψ

C21

C22

C2N

Ψ

CM1

CM2

CMN

Ψ

Decisio

n

Activity level

(base classifier)

Sensor level

(sensor classifier)

Network level

(sensor fusion)

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

Sensor 1

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

A Novel Method: Hierarchical Weighted Classifier N activities & M sensors

S1 α11

Ψ

C12

C1N

C11 β11

α12 β12

α1N β1N

S1 γ11,…,1N δ11,…,1N

𝑊𝐷𝑚𝑛 𝑞𝑚 𝑥𝑚𝑘=

𝛾𝑚𝑛, 𝑞𝑚 𝑥𝑚𝑘= 𝑛

𝛿𝑚𝑛 , 𝑞𝑚 𝑥𝑚𝑘≠ 𝑛

𝛾𝑚 = 𝛾𝑚1, 𝛾𝑚2, … , 𝛾𝑚𝑛 =𝑇𝑃𝑚1

𝑇𝑃𝑚1 + 𝐹𝑁𝑚1,

𝑇𝑃𝑚2

𝑇𝑃𝑚2 + 𝐹𝑁𝑚2, … ,

𝑇𝑃𝑚𝑛

𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛

𝑞𝑚 𝑥𝑚𝑘= 𝑎𝑟𝑔𝑚𝑎𝑥

𝑞

𝑂𝑚 𝑥𝑚𝑘

𝑂𝑚 𝑥𝑚𝑘= 𝑊𝐷𝑚𝑛 𝑥𝑚𝑘

𝑁

𝑛=1

𝛼𝑚𝑛 =𝑇𝑃𝑚𝑛

𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛

𝛽𝑚𝑛 =𝑇𝑁𝑚𝑛

𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛

𝑊𝐷𝑚𝑛 𝑥𝑚𝑘=

𝛼𝑚𝑛, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

0, 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

∀𝑞 = 𝑛

𝛽𝑚𝑛 , 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

0, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

∀𝑞 ≠ 𝑛

𝛿𝑚 = 𝛿𝑚1, 𝛿𝑚2, … , 𝛿𝑚𝑛 =𝑇𝑁𝑚1

𝑇𝑁𝑚1 + 𝐹𝑃𝑚1,

𝑇𝑁𝑚2

𝑇𝑁𝑚2 + 𝐹𝑃𝑚2, … ,

𝑇𝑁𝑚𝑛

𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛

∀𝑛 = 1, … , 𝑁

34

Sensor M

Sensor 2

SM

S2

S1 α11

Ψ

C12

C1N

C11

Ψ

C21

C22

C2N

Ψ

CM1

CM2

CMN

Ψ

Decisio

n

Activity level

(base classifier)

Sensor level

(sensor classifier)

Network level

(sensor fusion)

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

Sensor 1

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

A Novel Method: Hierarchical Weighted Classifier

S1 α11

Ψ

C12

C1N

C11 β11

α12 β12

α1N β1N

S1 γ11,…,1N δ11,…,1N

Ψ

Decisio

n

N activities & M sensors

𝑞 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑞

𝑂 𝑥𝑚

𝑂 𝑥𝑚 = 𝑂 𝑥1𝑘, 𝑥2𝑘, … , 𝑥𝑀𝑘= 𝑊𝐷𝑝 𝑞𝑝 𝑥𝑝𝑘

𝑀

𝑝=1

𝑊𝐷𝑚𝑛 𝑞𝑚 𝑥𝑚𝑘=

𝛾𝑚𝑛, 𝑞𝑚 𝑥𝑚𝑘= 𝑛

𝛿𝑚𝑛 , 𝑞𝑚 𝑥𝑚𝑘≠ 𝑛

𝛾𝑚 = 𝛾𝑚1, 𝛾𝑚2, … , 𝛾𝑚𝑛 =𝑇𝑃𝑚1

𝑇𝑃𝑚1 + 𝐹𝑁𝑚1,

𝑇𝑃𝑚2

𝑇𝑃𝑚2 + 𝐹𝑁𝑚2, … ,

𝑇𝑃𝑚𝑛

𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛

𝑞𝑚 𝑥𝑚𝑘= 𝑎𝑟𝑔𝑚𝑎𝑥

𝑞

𝑂𝑚 𝑥𝑚𝑘

𝑂𝑚 𝑥𝑚𝑘= 𝑊𝐷𝑚𝑛 𝑥𝑚𝑘

𝑁

𝑛=1

𝛼𝑚𝑛 =𝑇𝑃𝑚𝑛

𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛

𝛽𝑚𝑛 =𝑇𝑁𝑚𝑛

𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛

𝑊𝐷𝑚𝑛 𝑥𝑚𝑘=

𝛼𝑚𝑛, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

0, 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

∀𝑞 = 𝑛

𝛽𝑚𝑛 , 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

0, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞

∀𝑞 ≠ 𝑛

𝛿𝑚 = 𝛿𝑚1, 𝛿𝑚2, … , 𝛿𝑚𝑛 =𝑇𝑁𝑚1

𝑇𝑁𝑚1 + 𝐹𝑃𝑚1,

𝑇𝑁𝑚2

𝑇𝑁𝑚2 + 𝐹𝑃𝑚2, … ,

𝑇𝑁𝑚𝑛

𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛

∀𝑛 = 1, … , 𝑁

Evaluation of the Tolerance to Sensor Technological Anomalies

• Model validation

– Performance in ideal circumstances

– Tolerance to sensor failures

– Tolerance to sensor faults

35

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

• Benchmark dataset:

– MIT Activities of Daily Living Dataset*

• 9 acts

• 5 biaxial accelerometers

• 20 subjects (17-48 years old)

• Out-of-lab

• Experimental setup:

36

Five

acc

eler

om

eter

s

Walking Sitting and relaxing Standing still Running Bicycling Lying down Brushing teeth Climbing stairs

36

Eig

ht

act

ivit

ies

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

5 biaxial accelerometers

(limbs and trunk)

LP Elliptic Filter (Fc = 20Hz)

6 seconds sliding window

Mean, STD, kurtosis, MCR,

(...)

DT, NB, KNN, SVM (as base

classifiers)

Evaluation of the Tolerance to Sensor Technological Anomalies

* http://architecture.mit.edu/house_n/data/Accelerometer/BaoIntille.htm

• Performance in ideal circumstances:

37

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

Evaluated models: - SARC (≡ S) - FFMARC (≡ FF) - HWC

Parameters: - Feature sets: 1, 5,

10, 20 feat. - Base classifiers: DT,

NB, KNN, SVM Evaluation procedure: - 10-fold CV - 100 iterations

Las figuras irán a color

• Performance in ideal circumstances:

38

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

Evaluated models: - SARC (≡ S) - FFMARC (≡ FF) - HWC

Parameters: - Feature sets: 1, 5,

10, 20 feat. - Base classifiers: DT,

NB, KNN, SVM Evaluation procedure: - 10-fold CV - 100 iterations

Las figuras irán a color

• Performance in ideal circumstances:

39

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

Evaluated models: - SARC (≡ S) - FFMARC (≡ FF) - HWC

Parameters: - Feature sets: 1, 5,

10, 20 feat. - Base classifiers: DT,

NB, KNN, SVM Evaluation procedure: - 10-fold CV - 100 iterations

Las figuras irán a color

• Tolerance to sensor failures:

40

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

Evaluated model: - HWC

Parameters: - Feature set: 10

feat. - Classifier: KNN

Evaluation procedure: - 10-fold CV - 100 iterations

Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)

Baseline Accuracy (Ideal conditions) =

96.34%

• Tolerance to sensor failures:

41

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

1 missing sensor

Evaluated model: - HWC

Parameters: - Feature set: 10

feat. - Classifier: KNN

Evaluation procedure: - 10-fold CV - 100 iterations

Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)

Baseline Accuracy (Ideal conditions) =

96.34%

• Tolerance to sensor failures:

42

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

1 missing sensor

2 missing sensors

Evaluated model: - HWC

Parameters: - Feature set: 10

feat. - Classifier: KNN

Evaluation procedure: - 10-fold CV - 100 iterations

Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)

Baseline Accuracy (Ideal conditions) =

96.34%

• Tolerance to sensor failures:

43

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

1 missing sensor

2 missing sensors

3 missing sensors

4 missing sensors

Evaluated model: - HWC

Parameters: - Feature set: 10

feat. - Classifier: KNN

Evaluation procedure: - 10-fold CV - 100 iterations

Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)

Baseline Accuracy (Ideal conditions) =

96.34%

• Tolerance to sensor faults:

44

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

Ideal case Dynamic range shortening

• Tolerance to sensor faults:

45

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Evaluation of the Tolerance to Sensor Technological Anomalies

Evaluated models: - SARC - FFMARC - HWC

Parameters: - Feature set: 10 feat. - Classifier: KNN Evaluation procedure: - 10-fold CV - 100 iterations

AR model/ #faulty sensors 0 1 2 3 4 5

New dynamic range = 30% original dynamic range [-3g,3g]

SARC (hip) 82±5 66±4 - - - - SARC (wrist) 88±5 54±6 - - - - SARC (arm) 80±3 58±7 - - - -

SARC (ankle) 83±4 58±8 - - - - SARC (thigh) 89±2 72±4 - - - -

FFMARC 97±2 88±4 76±5 61±8 42±11 39±13 HD 90±3 85±4 80±9 68±13 59±16 53±20 MV 82±6 79±5 67±7 43±10 36±14 31±19

HWC 96±2 96±2 93±3 86±5 73±8 65±14 New dynamic range = 10% original dynamic range [-1g,1g]

SARC (hip) 82±5 21±11 - - - - SARC (wrist) 88±5 18±9 - - - - SARC (arm) 80±3 26±14 - - - -

SARC (ankle) 83±4 21±7 - - - - SARC (thigh) 89±2 20±6 - - - -

FFMARC 97±2 70±5 41±8 17±15 21±11 18±9 HD 90±3 80±6 59±13 42±12 30±17 21±16 MV 82±6 77±6 46±11 38±10 27±13 26±8

HWC 96±2 94±2 87±6 53±2 27±17 25±19

Conclusions

• Assuming a lifelong invariant sensor setup is unrealistic and may lead to a malfunctioning of the activity recognition system

• Body-worn sensors are subject to faults (signal degradation) and failures (absence of signal) normally unforeseen at design and runtime

• Classic activity recognition approaches (SARC, FFMARC) are not capable of dealing with sensor failures and are of limited utility under the effect of sensor faults

• The proposed alternate model (HWC) renders similar performance to standard activity recognition models in ideal conditions, proves to be robust to sensor failures and a relevant tolerance to sensor faults

46

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Robustness of AR systems to sensor deployment variations

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Objective: “Research the robustness of standard AR systems to unforeseen variations in the sensor deployment, as well as contribute with an alternate

approach to cope with these practical anomalies”

Problem Statement

48

SENSOR DEPLOYMENT CHANGES

Are activity recognition systems flexible enough to

allow users to wear the sensors on their own?

Is it possible to keep the systems functioning under the effects of sensor displacement?

Activity recognition process

Body motion sensing

Signal processing and

reasoning

Recognition of activities

Body motion sensing

Signal processing and

reasoning

Recognition of activities

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor Displacement

• Categories of sensor displacement

– Static: position changes can remain static across the execution of many activity instances, e.g. when sensors are attached with a displacement each day

– Dynamic: effect of loose fitting of the sensors, e.g. when embedded into clothes

• Sensor displacement new sensor position signal space change

• Sensor displacement effects depends on

– Original/end position and body part

– Activity/gestures/movements performed

– Sensor modality

49

Sensor displacement = rotation + translation (angular displacement) (linear displacement)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor Displacement Effects

Changes in the signal space propagates through the activity recognition chain (e.g., variations in the feature space)

RCIDEAL LCIDEAL= LCSELF

50

RCSELF ≠ RCIDEAL

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor Displacement in AR: Related Work

• Features invariant to sensor displacement

– Heuristics (Kunze08)

– Genetic algorithm for feature selection (Förster09a)

• Feature distribution adaptation

– Covariate shift unsupervised adaptation (Bayati09)

– Online-supervised user-based calibration (Förster09b)

• Classification (dis)similarity

– Output classifiers correlation (Sagha11)

K. Kunze and P. Lukowicz. Dealing with sensor displacement in motion-based onbody activity recognition systems. In 10th international conference on Ubiquitous computing, pp. 20–29, 2008.

K. Förster, P. Brem, D. Roggen, and G. Tröster. Evolving discriminative features robust to sensor displacement for activity recognition in body area sensor networks. In Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP), 2009 5th International Conference on, pp. 43–48, 2009.

H. Bayati, J. del R Millan, and R. Chavarriaga. Unsupervised adaptation to on-body sensor displacement in acceleration-based activity recognition. In Wearable Computers (ISWC), 2011 15th Annual International Symposium on, pp. 71–78, June 2011.

K. Förster, D. Roggen, and G. Tröster. Unsupervised classifier self-calibration through repeated context occurrences: Is there robustness against sensor displacement to gain? In Proc. 13th IEEE Int. Symposium on Wearable Computers (ISWC), pp. 77–84, 2009.

H. Sagha, J. R. del Millán, and R. Chavarriaga. Detecting and rectifying anomalies in Opportunistic sensor networks 8th Int. Conf. on Networked Sensing Systems, pp. 162-–167, 2011

51

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Approaches to Investigate on Sensor Displacement

El enfoque debería ser más bien… ahora vamos a comprobar la validez de nuestro modelo para sensor displacement, de modo que las dos siguientes transparencias se dejarían fuera…

Synthetically Modeled

Sensor Displacement

Realistic Sensor

Displacement

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Cambiar el título a «Sensor Displacement Study» o algo así…

52

Approaches to Investigate on Sensor Displacement

El enfoque debería ser más bien… ahora vamos a comprobar la validez de nuestro modelo para sensor displacement, de modo que las dos siguientes transparencias se dejarían fuera…

Synthetically Modeled

Sensor Displacement

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

53

Synthetically Modeled Sensor Displacement

• Sensor rotation Rotational noise (RN)

• Sensor translation Additive noise (AN)

• Examples:

54

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

𝑀𝑅𝑁 =

𝑐 𝜃 𝑐 𝜓 −𝑐 𝜙 𝑠 𝜓 + 𝑠 𝜙 𝑠 𝜃 𝑐 (𝜓) 𝑠 𝜙 𝑠 𝜓 + 𝑐 𝜙 𝑠 𝜃 𝑐 (𝜓)

𝑐 𝜃 𝑠 𝜓 𝑐 𝜙 𝑐 𝜓 + 𝑠 𝜙 𝑠 𝜃 𝑠 (𝜓) −𝑠 𝜙 𝑐 𝜓 + 𝑐 𝜙 𝑠 𝜃 𝑠(𝜓)

− 𝑠 𝜃 𝑠 𝜙 𝑐 𝜃 𝑐 𝜙 𝑐 𝜃

𝑥𝑟𝑜𝑡𝑦𝑟𝑜𝑡𝑧𝑟𝑜𝑡

= 𝑀𝑅𝑁 ×

𝑥𝑟𝑎𝑤𝑦𝑟𝑎𝑤𝑧𝑟𝑎𝑤

𝑥𝑡𝑟𝑦𝑡𝑟𝑧𝑡𝑟

= 𝑇𝐴𝑁 +

𝑥𝑟𝑎𝑤𝑦𝑟𝑎𝑤𝑧𝑟𝑎𝑤

𝑇𝐴𝑁 = 𝜇𝐴𝑁 + 𝜎𝐴𝑁2 + 𝑟𝑎𝑛𝑑_𝑛𝑜𝑟𝑚𝑎𝑙_𝑑𝑖𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛 (𝜇𝐴𝑁=0)

0 1 2 3

-2

0

2

Acce

lera

tio

n (

g)

Time (s)

Original

0 1 2 3

-2

0

2

Time (s)

RN

=15º

0 1 2 3

-2

0

2

Time (s)

RN

=90º

0 1 2 3

-2

0

2

Time (s)

AN

=0.1g

0 1 2 3

-2

0

2

Time (s)

AN

=0.5g

0 1 2 3

-2

0

2

Acce

lera

tio

n (

g)

Time (s)

Original

0 1 2 3

-2

0

2

Time (s)

RN

=15º

0 1 2 3

-2

0

2

Time (s)

RN

=90º

0 1 2 3

-2

0

2

Time (s)

AN

=0.1g

0 1 2 3

-2

0

2

Time (s)

AN

=0.5g

0 1 2 3

-2

0

2

Acce

lera

tio

n (

g)

Time (s)

Original

0 1 2 3

-2

0

2

Time (s)

RN

=15º

0 1 2 3

-2

0

2

Time (s)

RN

=90º

0 1 2 3

-2

0

2

Time (s)

AN

=0.1g

0 1 2 3

-2

0

2

Time (s)

AN

=0.5g

0 1 2 3

-2

0

2

Acce

lera

tio

n (

g)

Time (s)

Original

0 1 2 3

-2

0

2

Time (s)

RN

=15º

0 1 2 3

-2

0

2

Time (s)

RN

=90º

0 1 2 3

-2

0

2

Time (s)

AN

=0.1g

0 1 2 3

-2

0

2

Time (s)

AN

=0.5g

Walking Sitting

Proposed in: H. Sagha, J. R. del Millán, and R. Chavarriaga. Detecting and rectifying anomalies in Opportunistic sensor networks. 8th Int. Conf. on Networked Sensing Systems, pp. 162 – 167, 2011

• Benchmark dataset:

– MIT Activities of Daily Living Dataset*

• 9 acts

• 5 biaxial accelerometers

• 20 subjects (17-48 years old)

• Out-of-lab

• Experimental setup:

55

Five

acc

eler

om

eter

s

Walking Sitting and relaxing Standing still Running Bicycling Lying down Brushing teeth Climbing stairs

55

Eig

ht

act

ivit

ies

Evaluation of the Robustness to Sensor Displacement (Synthetic)

* http://architecture.mit.edu/house_n/data/Accelerometer/BaoIntille.htm INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

5 biaxial accelerometers

(limbs and trunk)

LP Elliptic Filter (Fc = 20Hz)

6 seconds sliding window

10 feat. set KNN (as base

classifier)

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

• Performance drop under the effects of sensor rotation and translation:

56

Evaluated model: - SARC Parameters: - Feature set: 10 feat. - Base classifiers: KNN Evaluation procedure: - 10-fold CV - 100 iterations

Evaluation of the Robustness to Sensor Displacement (Synthetic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Ro

tati

on

Tr

ansl

atio

n

• Performance drop under the effects of sensor rotation and translation:

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

57

Evaluated model: - SARC Parameters: - Feature set: 10 feat. - Base classifiers: KNN Evaluation procedure: - 10-fold CV - 100 iterations

Evaluation of the Robustness to Sensor Displacement (Synthetic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

30%

Ro

tati

on

Tr

ansl

atio

n

23% 3%

8%

8%

20%

15% 25% 5%

15%

1%

4%

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

• Performance drop under the effects of sensor rotation and translation:

58

Evaluated model: - SARC Parameters: - Feature set: 10 feat. - Base classifiers: KNN Evaluation procedure: - 10-fold CV - 100 iterations

Evaluation of the Robustness to Sensor Displacement (Synthetic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

70% Ro

tati

on

Tr

ansl

atio

n

50% 20%

55%

6%

15% 3%

8%

8%

20%

15% 45%

75%

5%

15% 43%

30% 1%

4%

4%

10% 25%

30%

23%

Approaches to Investigate on Sensor Displacement

Realistic Sensor

Displacement

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

59

• No dataset for studying the effects of sensor displacement!

• Observe

– Variability introduced with respect to the ideal setup when the sensors are self-placed by the users

– Effects of large sensor displacements (extreme de-positioning)

• Scenarios

– Ideal-placement

– Self-placement

– Induced-displacement

Implementing Realistic Sensor Displacement

Ideal Self Induced

60

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

NEW DATASET* (REALDISP)

*Freely available at: www.ugr.es/~oresti/datasets

REALDISP Dataset: Study Setup

• Cardio-fitness room

• 9 IMUs (9DoF) ACC, GYR, MAG

• Laptop data storage and labeling*

• Camera offline data validation

• 17 volunteers (22-37 years old)

*Annotation tool: http://crnt.sourceforge.net/CRN_Toolbox/Home.html 61

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

REALDISP Dataset: Activity Set

• Activities intended for:

– Body-general motion: Translation | Jumps | Fitness

– Body-part-specific motion: Trunk | Upper-extremities | Lower-extremities

62

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

• Experimental setup:

• Studies:

– AR systems: SARC, FFMARC, HWC

– Settings: Ideal-placement, Self-placement, Induced-displacement

– Scenarios: 10 activities, 20 activities, 33 activities (all)

• Evaluation procedure

– 10-fold CV, 100 iterations

63

Evaluation of the Robustness to Sensor Displacement (Realistic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

9 triaxial accelerometers (all limbs and

trunk)

No preprocessing

(raw data)

6 seconds sliding window

FS1=mean

FS2=mean,std

FS3=mean,std,max,min,mcr

DT, KNN, NB (as base classifiers)

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

64

Evaluation of the Robustness to Sensor Displacement (Realistic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Idea

l Se

lf

Ind

uce

d

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

65

Evaluation of the Robustness to Sensor Displacement (Realistic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Idea

l Se

lf

Ind

uce

d

13%

25%

25%

45%

3%

13%

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

66

Evaluation of the Robustness to Sensor Displacement (Realistic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Idea

l Se

lf

Ind

uce

d

13%

25%

15%

50%

25%

45%

30%

45%

3%

13%

5%

15%

HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)

67

Evaluation of the Robustness to Sensor Displacement (Realistic)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Idea

l Se

lf

Ind

uce

d

13%

25%

15%

50% 50%

15% 25%

45%

30%

45% 45%

30%

3%

13%

5%

15%

5%

25%

Conclusions

• Classic activity-aware systems assume a predefined sensor deployment that further remains unchanged during runtime, which are not lifelike assumptions

• Body-worn inertial sensors are subject to deployment changes (displacement) in real-world contexts, potentially leading to signal variations with respect to ideal patterns

• Activity recognition systems proves to be more sensitive to sensor rotations than translations, specially when located on body parts of reduced mobility

• Standard models (SARC,FFMARC) suffer from a critical performance worsening when the sensors are largely depositioned or self-placed by the users

• The HWC significantly outperforms the tolerance of standard activity recognition models (up to 30%), effectively showing outstanding capabilities to assimilate the changes introduced during the self-placement of the sensors and to moderately overcome the situation of largely depositioned sensors

68

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Hablar de realistic, novel, innovatively… Use text below…

Supporting AR systems network changes: instruction of

newcomer sensors

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Objective: “Study the capacity of standard AR systems to support unforeseen changes in the sensor network, as well as contribute with an

alternate approach to cope with these topological variations”

Problem Statement

70

SENSOR INFRASTRUCTURE CHANGES

- Sensor replacement, buys a new gadget, uses a different device depending on the context

- System without recognition capabilities, or at least not the ones desired onwards…

- New dataset must be collected, user involvement,… ?

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Collect a training dataset

Train and test the model

The AR system is “ready”

Collect a training dataset

Train and test the model

The AR system is “ready”

Do we need to collect a new dataset each time

the sensor topology changes?

Is it possible to leverage the knowledge of a functional system to instruct a system to

operate on a newcomer sensor?

Activity recognition system design

Infraestructure Changes: Newcomer Sensors

71

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor replacement (repair/upgrade)

Sensor addition (redundancy)

Sensor discovery (opportunistic use)

Transfer learning

Instruction of Newcomer Sensors

72

Teacher

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Classic approach

Limitations: - Predefined setup and deployments - System designer involvement - User/s involvement

Learner

“Mechanism, ability or means to recognize and apply knowledge and skills learned in previous tasks or domains to novel tasks or domains”

Collection of a new dataset for each possible scenario

Transfer Learning in AR: Related Work

• Transfer between wearable sensors

– Translation of locomotion recognition capabilities (Calatroni11)

• Model parameters

• Labels

• Transfer between ambient sensors

– Translation among smart homes through meta-featuring (van Kasteren10)

• Common meta-feature space

• Limitations

– Long time scales operation

– Incomplete transfer

– Difficult transfer across modalities

A. Calatroni, D. Roggen, and G. Tröster, “Automatic transfer of activity recognition capabilities between body-worn motion sensors: Training newcomers to recognize locomotion,” in Proc. 8th Int Conf on Networked Sensing Systems, 2011. T. van Kasteren, G. Englebienne, and B. Kröse, “Transferring knowledge of activity recognition across sensor networks,” in Proc. 8th Int. Conf on Pervasive Computing, 2010.

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

73

Multimodal Transfer Methods

• System identification (signal level)

• Transfer methods (reasoning level)

Ψ𝐴→𝐵 𝑡 Sensor Domain

A

Sensor Domain

B

Activity Templates

Activity Models

0 1 2 3-0.5

0

0.5

1

1.5

Time (s)

Accele

ration (

G)

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of activity models (features + labels, classification models)

Transfer of activity templates (patterns + labels)

74

Transfer of Activity Templates

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

• Transfer of the recognition capabilities of an existing source system (S) that operates on activity templates (patterns) to an untrained target system (T) that lacks from these capabilities

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

75

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

Coexistence… (T)

0 20 40-1

0

1

2

Time (s)

Positio

n (

m)

0 20 40-1

0

1

2

Time (s)A

ccele

ration (

G)

Transfer of Activity Templates

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

(1) Both systems coexists during a certain period of time

76

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

(2) A mapping function between source and target domains is discovered through system identification (MIMO model)

Transfer of Activity Templates

Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)

77

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

(3) The activity templates are translated from source to target domain

Transfer of Activity Templates

78

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Templates

(3) The activity templates are translated from source to target domain

System S (source domain) System T (target domain)

L3

79

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)

0 1 2 3-0.5

0

0.5

1

1.5

Time (s)

Accele

ration (

G)

X

Y

Z

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Templates

(3) The activity templates are translated from source to target domain

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

L3 L3

80

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

L1 L2 L3

Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Templates

• System identification

– 3) The activity templates are translated from source to target domain

0 1 2 3

0

0.5

1

1.5

Time (s)

Accele

ration (

G)

XYZ

X

Y

ZFitness.. Esto es lo que idealmente busco…

81

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

L1 L2 L3 0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Templates

(4) Once the templates have been translated, the target system is ready for activity detection

82

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

L1 L2 L3 0 2 4

-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

L1 L2 L3 0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Templates

(4) Once the templates have been translated, the target system is ready for activity detection

Instruction completed!

83

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

• Transfer of the recognition capabilities of an existing source system (S) that operates on activity models (features + classification model) to an untrained target system (T) that lacks from these capabilities

84

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

Coexistence… (T)

0 20 40-1

0

1

2

Time (s)P

ositio

n (

m)

0 20 40-1

0

1

2

Time (s)

Accele

ration (

G)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

(1) Both systems coexists during a certain period of time

85

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

(2) A mapping function between target and source domains is discovered through system identification (MIMO model)

86

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

(3) The source activity models are translated to the target domain so both use the same activity models

87

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

(3) The source activity models are translated to the target domain so both use the same activity models; these activity models also define the target activity recognition system

88

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

(4) The target system continuously translate its signals into the source domain to operate on the transferred recognition system

0 1 2 3 4-0.5

0

0.5

1

1.5

XYZ

89

𝑋𝑆(𝑡)

𝑋𝑇(𝑡)

System S (source domain) System T (target domain)

Sig

na

l le

vel

Rea

son

ing

le

vel

Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Transfer of Activity Models

(4) The target system continuously translate its signals into the source domain to operate on the transferred recognition system; since then it is ready for activity detection

Instruction completed!

0 1 2 3 4-1

0

1

2

X

Y

Z

90

Evaluation of Multimodal Transfer

• Models validation

– Transfer between IMU and IMU (Identical Domain Transfer)

– Transfer between Kinect and IMU (Cross Domain Transfer)

91

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

L1 L2 L3 0 2 4 6

-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4 6-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 2 4-1

0

1

2

Time (s)

Positio

n (

m)

XYZ

0 1 2 3-1

0

1

2

Time (s)

Positio

n (

m)

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

L1 L2 L3 0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 5 10-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

0 2 4-1

0

1

2

Time (s)

Accele

ration (

G)

X

Y

Z

Transfer of Activity Templates

Transfer of Activity Models

0 1 2 3 4-1

0

1

2

X

Y

Z

Multimodal Kinect-IMU Dataset: Study Setup

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

*Freely available at: www.ugr.es/~oresti/datasets 92

Multimodal Kinect-IMU Dataset: Study Setup

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

MTx XSENS IMUs

- 3D ACC + (3D GYR, 3D MAG, 4D QUA) - Sampling rate 30Hz

Applications

93 Xsens data logger http://crnt.sourceforge.net/CRN_Toolbox/References.html

Multimodal Kinect-IMU Dataset: Study Setup

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

MICROSOFT KINECT

- RGB cam + IR cam + IR led - Depth map (0.5-6m)

- 15 joints skeleton tracking - 3D position - Tracking range (1.2-3.5m) - Sampling rate 30Hz

Applications

94 Kinect data logger http://code.google.com/p/qtkinectwrapper/

Multimodal Kinect-IMU Dataset: Scenarios

Geometric Gestures (HCI)

48 instances per gesture

Other scenarios were also collected as part of this dataset (more info at www.ugr.es/~oresti/datasets) 95

Multimodal Kinect-IMU Dataset: Scenarios

Geometric Gestures (HCI) Idle (Background)

~5 min of data 48 instances per gesture

Other scenarios were also collected as part of this dataset (more info at www.ugr.es/~oresti/datasets) 96

Transfer between IMU and IMU

• Analyzed transfers

– Transfer of Activity Templates and Activity Models from:

• RLA (3D acceleration) to RUA (3D acceleration)

• RUA (3D acceleration) to RLA (3D acceleration)

• RUA (3D acceleration) to BACK (3D acceleration)

• BACK (3D acceleration) to RUA (3D acceleration)

• RLA (3D acceleration) to BACK (3D acceleration)

• BACK (3D acceleration) to RLA (3D acceleration)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

97

Evaluation of Transfer between IMU and IMU

• Mapping:

– Model MIMO3x3 mapping with 10 tap delay

– Types

• Problem-domain mapping (PDM)

• Gesture-specific mapping (GSM)

• Unrelated-domain mapping (UDM)

– Learning 100 samples (~3.3s)

• Activity recognition model:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Triaxial acceleration

(IMU)

No preprocessing

(raw data)

Instance based segmentation

FS=max,min KNN (standard

classifier)

98

Evaluation of Transfer between IMU and IMU

• Transfer of Activity Templates:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

99 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

LA=lower arm UA=upper arm

B=back

Evaluation of Transfer between IMU and IMU

• Transfer of Activity Templates:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

100 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

LA=lower arm UA=upper arm

B=back

<1% <3%

Evaluation of Transfer between IMU and IMU

• Transfer of Activity Templates:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

101 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

LA=lower arm UA=upper arm

B=back

<1% <3%

35%

15% 17%

28% 35% 28%

10% 10%

Evaluation of Transfer between IMU and IMU

• Transfer of Activity Templates:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

102 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

LA=lower arm UA=upper arm

B=back

<1% <3%

12% 20%

35%

55%

15% 17%

28% 35%

60%

28%

30%

50%

10% 10%

Evaluation of Transfer between IMU and IMU

• Transfer of Activity Models:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

103 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

LA=lower arm UA=upper arm

B=back

Evaluation of Transfer between IMU and IMU

• Transfer of Activity Models:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

104 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

LA=lower arm UA=upper arm

B=back

Transfer between Kinect and IMU

• Analyzed transfers

– Transfer of Activity Templates (Kinect to IMU) :

• HAND (3D position) RLA (3D acceleration)

• HAND (3D position) RUA (3D acceleration)

• HAND (3D position) BACK (3D acceleration)

– Transfer of Activity Models (IMU to Kinect):

• RLA (3D acceleration) HAND (3D position)

• RUA (3D acceleration) HAND (3D position)

• BACK (3D acceleration) HAND (3D position)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

105

Evaluation of Transfer between Kinect and IMU

• Mapping:

– Model MIMO3x3 mapping with 10 tap delay

– Types

• Problem-domain mapping (PDM)

• Gesture-specific mapping (GSM)

• Unrelated-domain mapping (UDM)

– Learning 100 samples (~3.3s)

• Activity recognition model:

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

106

Triaxial acceleration

(IMU) / Triaxial position (KINECT)

No preprocessing

(raw data)

Instance based segmentation

FS=max,min KNN (standard

classifier)

Evaluation of Transfer between Kinect and IMU

• Transfer of Activity Templates (From Kinect to IMU)

• Transfer of Activity Models (From IMU to Kinect)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

107 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

RLA= right lower arm RUA= right upper arm

BACK=back KINECT=hand

Evaluation of Transfer between Kinect and IMU

• Transfer of Activity Templates (From Kinect to IMU)

• Transfer of Activity Models (From IMU to Kinect)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

108 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

RLA= right lower arm RUA= right upper arm

BACK=back KINECT=hand

<4% <4% <8% <6%

Evaluation of Transfer between Kinect and IMU

• Transfer of Activity Templates (From Kinect to IMU)

• Transfer of Activity Models (From IMU to Kinect)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

109 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

RLA= right lower arm RUA= right upper arm

BACK=back KINECT=hand

<4% <4% <8% <6%

30%

45%

35%

60%

Evaluation of Transfer between Kinect and IMU

• Transfer of Activity Templates (From Kinect to IMU)

• Transfer of Activity Models (From IMU to Kinect)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

110 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping

RLA= right lower arm RUA= right upper arm

BACK=back KINECT=hand

<4%

35%

50%

<4% <8% <6%

30%

45%

35%

60% 55%

30% 35% 35%

Evaluation of Transfer between Kinect and IMU

• Transfer of Activity Templates (From Kinect to IMU)

• Transfer of Activity Models (From IMU to Kinect)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

From Kinect to IMU (RLA) From IMU (RLA) to Kinect

FS1=mean FS2=max,min

111 30 samples = 1 s

Evaluation of Transfer between Kinect and IMU

• Transfer of Activity Templates (From Kinect to IMU)

• Transfer of Activity Models (From IMU to Kinect)

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

From Kinect to IMU (RLA) From IMU (RLA) to Kinect

FS1=mean FS2=max,min

112 30 samples = 1 s

25% 20%

Conclusions

• Classical training procedures are not practical to instruct newcomer sensors in dynamically varying and evolvable activity recognition setups

• A novel multimodal transfer learning model is proposed to translate the recognition capabilities of an existing system to a new untrained system, at runtime and without expert or user intervention

• As few as a single gesture (≈3 seconds) of data is enough to learn a mapping model that captures the underlying relation between systems of identical or different modality

• The transfer between IMUs across close-by limbs achieves a recognition accuracy superior to 97% (>2% below baseline), and 95% (>4% below baseline) for the transfer between Kinect and IMU, independently of the direction of the transfer

• Low-variance data unrelated to the activities of interest can be also used to learn a mapping, albeit with more data

113

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Conclusions and future work

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Contributions

• Identification of the requirements and challenges posed by AR systems in real-world conditions

• Evaluation of the tolerance of standard AR systems to sensor technological anomalies, particularly sensor failures and faults

• Definition and development of a novel model, so-called HWC, to overcome the effects of sensor failures and faults. Evaluation of the robustness of the proposed HWC model to the effects of sensor failures and faults

• Evaluation of the tolerance of standard AR systems to sensor deployment variations, particularly static and dynamic sensor displacements

• Evaluation of the robustness of the proposed HWC model to the effects of sensor displacements

• Definition, development and validation of a novel multimodal transfer learning method that operates at runtime, with low overhead and without user or system designer intervention

115

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Contributions

• Collection and curation of an innovative benchmark dataset to investigate the effects of sensor displacement, introducing the concept of ideal-placement, self-placement and induced-displacement. This dataset includes a wide range of physical activities, sensor modalities and participants. Apart from investigating sensor displacement, the dataset lend itself for benchmarking activity recognition techniques in ideal conditions. The dataset is publicly available to the research community at http://www.ugr.es/~oresti/datasets

• Collection and curation of a novel multimodal dataset to investigate transfer learning among ambient sensing and wearable sensing systems. The dataset could be also used for gesture spotting and continuous activity recognition. The dataset is publicly available to the research community at http://www.ugr.es/~oresti/datasets

116

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Selected Publications

• International Journals (SCI-indexed) – Banos, O., Toth M. A., Damas, M., Pomares, H., Rojas, I. Dealing with the effects of

sensor displacement in wearable activity recognition. Sensors, MDPI (2014) [Under review]

– Banos, O., Damas, M., Guillen, A., Herrera, L.J., Pomares, H., Rojas, I. Multi-sensor fusion based on asymmetric decision weighting for robust activity recognition. Neural Processing Letters, Springer (2014) [Under review]

– Banos, O., Galvez, J. M., Damas, M., Pomares, H., Rojas, I. Window size impact in activity recognition. Sensors, MDPI, vol. 14, no. 4, pp. 6474-6499 (2014)

– Banos, O., Damas, M., Pomares, H., Rojas, F., Delgado-Marquez, B., Valenzuela, O. Human activity recognition based on a sensor weighting hierarchical classifier. Soft Computing, Springer, vol. 17, pp. 333-343 (2013)

– Banos, O., Damas, M., Pomares, H., Rojas, I. On the Use of Sensor Fusion to Reduce the Impact of Rotational and Additive Noise in Human Activity Recognition. Sensors, MDPI, vol. 12, no. 6, pp. 8039-8054 (2012)

– Banos, O., Damas, M., Pomares, H., Prieto, A., Rojas, I.: Daily Living Activity Recognition based on Statistical Feature Quality Group Selection. Expert Systems with Applications, Elsevier, vol. 39, no. 9, pp. 8013-8021 (2012)

117

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Selected Publications

• Book chapters – Banos, O., Toth M. A., Damas, M., Pomares, H., Rojas, I. Amft, O.: Evaluation of inertial sensor

displacement effects in activity recognition systems. Science and Supercomputing in Europe (Information & Communication Technologies), HPC-Europe 2 (2013) ISBN: 978-84-338-5400-1

• Conference papers – Banos, O., Damas, M., Pomares, H., Rojas, I.: Handling displacement effects in on-body sensor-

based activity recognition. In: Proceedings of the 5th International Work-conference on Ambient Assisted Living an Active Ageing (IWAAL 2013), San Jose, Costa Rica, December 2-6, (2013) [BEST PAPER AWARD]

– Banos, O., Damas, M., Pomares, H., Rojas, I.: Activity recognition based on a multi-sensor meta-classifier. In: Proceedings of the 2013 International Work Conference on Neural Networks (IWANN 2013), Tenerife, June 12-14, (2013)

– Banos, O., Toth, M. A., Damas, M., Pomares, H., Rojas, I., Amft, O.: A benchmark dataset to evaluate sensor displacement in activity recognition. In: Proceedings of the 14th International Conference on Ubiquitous Computing (Ubicomp 2012), Pittsburgh, USA, September 5-8, (2012)

118

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Selected Publications

• Conference papers (cont.) – Banos, O., Calatroni, A., Damas, M., Pomares, H., Rojas, I., Troester, G., Sagha, H., Millan, J. del

R., Chavarriaga, R., Roggen, D.: Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems Across Sensor Modalities. In: Proceedings of the 16th annual International Symposium on Wearable Computers (ISWC 2012), Newcastle, United Kingdom, June 18-22 (2012)

– Banos, O., Damas, M., Pomares, H., Rojas, I.: Human multisource activity recognition for AAL problems. In: Proceedings of the 5th International Symposium on Ubiquitous Computing and Ambient Intelligence (UCAmI 2011), Riviera Maya, Mexico, December 5-9, (2011)

– Banos, O., Damas, M., Pomares, H., Rojas, I.: Recognition of Human Physical Activity based on a novel Hierarchical Weighted Classification scheme. In: Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN 2011), IEEE, San Jose, California, July 31-August 5, (2011)

– Banos, O., Pomares, H., Rojas, I.: Ambient Living Activity Recognition based on Feature-set Ranking Using Intelligent Systems. In: Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN 2010), IEEE, Barcelona, July 18-23, (2010)

119

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Future Work

• Collection of new large standard datasets

• Dynamic reconfiguration of the HWC

• Self-adaptive HWC

• Tolerance to other sensor technological and topological anomalies

• Multiple trainers and complex modalities in transfer learning

• Integration in commercial systems and end-user applications

120

INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS

Sensor M

Sensor 2

SM

S2

S1 α11

Ψ

C12

C1N

C11

Ψ

C21

C22

C2N

Ψ

CM1

CM2

CMN

Ψ

Decisio

n

Activity level

(base classifier)

Sensor level

(sensor classifier)

Network level

(sensor fusion)

β11

α12 β12

α1N β1N

α21 β21

α22 β22

α2N β2N

αM1 βM1

αM2 βM2

αMN βMN

γ11,…,1N δ11,…,1N

γ21,…,2N δ21,…,2N

γM1,…,MN δM1,…,MN

Sensor 1

FEEDBACK

¡Gracias a todos!

121