Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Never Stand Still Faculty of Engineering Computer Science and Engineering

Click to edit Present’s Name

Never Stand Still Faculty of Engineering Computer Science and Engineering

KineticSense: Wearable Context Sensing

from Kinetic Energy Harvesting

Mahbub Hassan mahbub.hassan@unsw.edu.au

School of Computer Science and Engineering University of New South Wales, Sydney, Australia

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Health Care Problems in Australia

•  1993 to 2013, # of hospital beds dropped from 30 to 17 (per 1000 people of 65+)

•  2009 to 2013, private hospital admissions for subacute and non-acute services grew by 14% a year

•  2003 to 2013, health care cost grew from 68 B to 172 B (14% of GDP)

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Automating Health Care with Wearable Context Sensing

Health care provider

Patient@Home (No hospital admission)

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Growing Popularity of Wearable Devices

•  Continuous activity and context monitoring with cloud analytics •  Step count, calorie burning, sleep analysis, …

Sensor Sampling

Tx Sensor Data to Cloud

Cloud Analytic

1 2 3

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Battery Recharging Bottleneck •  Too many sensors to sample – accelerometers,

gyroscopes, magnetometers, microphones, GPS, temperature, humidity, pulse rate, …

•  Batteries last for few hours (intense use) to few weeks at most

•  Battery recharging is inconvenient at best and problematic in many cases, such as elderly patient monitoring

•  Battery recharging is a major roadblock for pervasive deployment of current wearable devices

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Why Not Use Human Energy Instead of Batteries? •  A human accumulates and expends a tremendous amount

of energy each day

1 doughnut = 1.3 Mega Joule

Sprinting expends 1.6KW

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Powering wearable devices from Kinetic Energy Harvesting convert motion energy to electrical energy

Piezoelectric Energy Harvester

Sensors

CPU

Radio

Harvested Power A few to few hundred µW could be harvested in a wrist-worn device

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Overview of My Talk ①  A new context sensing framework for human-powered

wearables à detect human contexts directly from harvested energy patterns à significantly more power efficient than existing sensor-based sensing paradigm

②  Experimental evidence of successful context sensing with real energy harvesters and real human subjects (Contexts: activity, steps, calorie, voice, mobility, location, identification)

③  Limitations of current prototype and future directions

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

The Essence of KineticSense

KEH Sensor Array (acc,gyro,mag,mic,…)

Context Detection

Power Samples

KEH Context Detection

Samples

KineticSense

Conventional Approach

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

EXPERIMENTING WITH REAL ENERGY HARVESTERS AND HUMANS

Can we infer human activity from kinetic energy harvesting?

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Piezoelectric Energy Harvester (vibration-based)

Mide.com

AC Rectifier

Recharge battery/

Capacitor

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

KEH Data Logger (samples AC voltage values)

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

VEH Data Logger

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #1 [IC2015] Human activity recognition – KEH Data Collection

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

KEH Time Series

Walking

Running

Standing

accelerometer VEH

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Does KEH contain information for activity recognition In

form

atio

n G

ain

Features

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Activity Recognition Accuracy – 5 activities (W, R, S, SU, SD)

Classifier Activity Recognition Accuracy (%)

Accelerometer-based

KEH-based

Hand Waist Hand Waist

K-nearest neighbour 95.01 98.70 81.13 87.01

Decision Trees 87.91 91.02 79.74 79.86

Multilayer Perceptron

88.25 96.39 78.29 85.52 Low

er th

an a

ccel

erom

eter

, bu

t not

too

bad

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #2 [iThings2015] Step counting

4 su

bjec

ts, s

trai

ght a

s w

ell

as tu

rnin

g pa

ths,

pea

k de

tect

ion

algo

rithm

, 570

st

eps,

96%

acc

urac

y

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #3 [BODYNET2015]

Calorie burning estimation •  10 subjects, 4 male, 6 female: waist mounted •  Anthropometric data: Age (26-35 years, mean = 29, s.d.

= 3.06), Weight (58-91 Kg, mean = 69.3 s.d. = 10.21), Height (154-185 cm, mean = 168.5, s.d. = 9.98)

•  Two activities: walking and running •  Linear regression model to estimate calorie burning from

energy harvesting samples and anthropometric data •  Leave-one-out cross validation (1 for testing, 9 for

training)

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Calorie estimation results - running

Close to accelerometer

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #4 [PerCom-WIP2016]

Mobility Monitoring

Train Car Bus Walking

Running

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

2-layer classification

Peak analysis

Mean analysis

Summary of Trace Data

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Results

93% 77% 88%

Bus and Car are confused, but not train

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #5 [WOWMOM2016]

Hotword detection

3cm

Quiet Room Conditions

Hotword: “OK Google” Non-hotwards: “Good morning”, “how are you”, “fine, thank you” 8 subjects: 4 m, 4 f 60 instances (30 hotword 30 non-hotword) per subject

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Speaker orientation

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Results

Flat Vertical

Speaker Independent

78% 62%

Speaker Dependent

85% 73%

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #6 [NDSS2017]

User Identification (via gait recognition)

Electromagnetic Energy Harvester (EEH)

We have experimented with both piezoelectric as well as electromagnetic energy harvesters

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

PEH and EEH Signals from Different Subjects (Walking)

2

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

User Identification (gait recognition) Accuracy

20 subjects

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Exp #7 [PerCom2017]

Location Context (based on short-range acoustic communication)

If the energy harvester can recognize known sound vibrations, then a wearable can detect its location from a sound beacon

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Impact of Transmitted Sound on Energy Harvester

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Communication Performance

5bps at 80 cm for BER < 1% (laptop to wearable)

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Natural Immunity Against Noise Even loud music didn’t affect the communication performance

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Conclusion

•  Positive: A vast array of useful contexts can be detected using KEH as a sensor à power saved can be used for collecting more physiological data (improved health care)

•  Negative: Power generated is small and detection accuracy not good for fine-grained contexts

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Future Directions

①  Advanced machine learning to improve context detection accuracy

②  Hybrid energy harvesters à increase both power and information content Ø  Multi-axial PEH, multi-modal KEH, hybrid with PEH+TEG

③  Fancy wireless communications, e.g., pulse-based energy rate transfer à reduce transmission power consumption [TOSN2017]

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

2-axial PEH prototype in progress

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

References 1.  [PerCom2015] Pervasive Self-powered Human Activity Recognition without the Accelerometer, PerCom 2015

2.  [IC2015] Energy-Harvesting Wearables for Activity-Aware Services, IEEE Internet Computing, 9(5), 2015 3.  [iThings2015] Step detection from power generation pattern in energy-harvesting wearable devices, IEEE iThings 2015

4.  [BODYNET2015] Estimating Calorie Expenditure from Output Voltage of Piezoelectric Energy Harvester - an Experimental Feasibility Study, BODYNETS 2015

5.  [PerCom_WIP2016] Transportation Mode Detection using Kinetic Energy Harvesting Wearables, PerCom Work-in-Progress, 2016

6.  [WOWMOM2016] Feasibility and Accuracy of Hotword Detection using Vibration Energy Harvester, WOWMOM 2016.

7.  [NDSS2017] KEH-Gait: Towards a Mobile Healthcare User Authentication System by Kinetic Energy Harvesting, NDSS 2017.

8.  [PerCom2017] VEH-COM: Demodulating Vibration Energy Harvesting for Short-Range Communications, PerCom 2017

9.  [TOSN2017] SEMON: Sensorless Event Monitoring in Self-powered Wireless Nanosensor Networks, ACM Tran. On Sensor Networks, Accepted 05 March 2017

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

Questions?

Mahbub Hassan Missouri University of Science and Technology, 10 March 2017

96% power saving

Power Measurements (KEH vs accelerometer) TI Sensor Tag