1

i-Care ProjectDietary-Aware Dining

Table: Observing Dietary Behavior

over Tabletop Surface

Keng-hao Chang, Shih-yen Liu, Toung Lin, Hao Chu,

Jane Hsu, Polly Huang, (Cheryl Chen)i-space Laboratory

National Taiwan University

2

What is it?

• A dietary-tracker built into an everyday dining table– Track what & how much you eat over tabletop

surface

• Motivation– We are what we eat– Food choices affect long-term & short-term

health

• Show a demo video

3

Smart Everyday Object

• Digital-enhanced everyday objects– Provide digital services

• Support natural human interactions– Natural human interactions = inputs to digital

services

• Goals – Providing digital services without (users)

operating digital devices → better usability– Human-centric computing: technology adapting

to users rather than users adapting & learning about technology

4

Outline for Reminder of Talk

• Related work• Approach• Assumptions & Limitations• Design & Implementation• Experimental Evaluation• Future work

5

Related Work

• Dietary trackers– Shopping receipt scanner (GaTec

h)– Chewing Sound (ETH)– My food phone (startup)

• Intelligent surfaces– Load sensing table (Lancester)– Smart floor (GaTech, NTU)– Posture Chair (MIT)

• What’s new here?– Accuracy– Fine-grained tracking– Simultaneous concurrent interac

tions

6

Contribution claims

• It is a fine-granularity (automated) dietary tracker.– It can track multiple concurrent

interactions from multiple individuals over the same tabletop surface.•People usually don’t eat alone

• It is an enhanced loading sensing table.

7

General Approach

• RFID tags on food containers• Two sensor surfaces on table

– Each surface is made of cells– RFID reader surface

• Detect RFID(s) in each cell– Weighting surface (load cells)

• Measure weight change in each cell• Track the food path from container(s) → conta

iner(s) → mouth using these two sensor surfaces

8

Assumptions (Limitations)

• Closed system rather than open system.– Food transfers among tabletop objects and

mouths, no external objects and food sources

• Users identified by personal containers (personal plates and cups)

• Food containers tagged with RFID tags• No cross-cell objects• No leaning their hands on the table• Not a mobile tracker

9

Single Interaction Example

• Bob pours tea from the tea pot to his personal cup, and drinks it

• Detect tea transfer from one container to another container1) Identify the presence & absence of containers

• RFID tags on containers• tag-food mapping

2) Track tea transfer • Weight change detection• Weight matching algorithm

10

Single Interaction Example

Pour tea?•Weight increases ∆w2.

• Bob pours tea from the tea pot to personal cup, and drinks it

Pick up tea pot.• RFID tag disappears• Weight decreases ∆w1

Put on tea pot.•RFID tag appears•Weight increases ∆w3 Pour tea!• |∆w3 - ∆w1 | ≈ ∆w2

11

Single Interaction Example

• Bob pours tea from the tea pot to personal cup, and drinks it

Pick up cup.• RFID tag disappears.• Weight decreases ∆w1.

Put on cup.•RFID tag appears.•Weight increases ∆w2.Drink tea? (only if no match)• Amount | ∆w2 - ∆w1 |

12

Concurrent Interactions Example

• Bob pours tea & Alice cuts cake

Pour tea?Cut cake? • Weight change ∆w

Pour tea• Weight increases ∆ w1

Cut cake• Weight decreases ∆w2

13

Concurrent Interactions Example

• Multiple, concurrent person-object interactions– The larger the cell, the higher the

possibility of concurrent interactions over a cell

– Cell size = average size of container– Reduce the possibility of concurrent

interactions over one cell

14

Design Architecture

Tag-object

mappings

Behavior Inference Engine

Event Interpreter

Weight Change Detector

Object Presence Detector

Weighing surface (weighing sensors)

RFID Surface (readers)

Applications (Dietary-aware Dining Table)

Common sense

semantics

Sensor Events

Intermediate Events

Dietary Behaviors

15

Inference Rule

Dietary behaviors

Behavior Inference Rules

Transfer(u, w, type)

Weight-Change(Object-i1, Δw1) ∩ (Δw1< 0) ∩ Weight- Change (Object-i2, Δw2) ∩ (Δw2 > 0) ∩ Contains(Object-i1, type) ∩ Owner(Object-i2, u) ∩(|Δw1 +Δ w2 |< ε) → Transfer (u, Δw2, type)

Eat(u, w, type) Weight-Change(Object-i, Δw) ∩ (Δw<0) ∩ Contains(Object-i, type) ∩ Owner(Object-i, u)→ Eat(u, -Δw, type)

16

Experimental setup

• 2 Dining settings– Afternoon tea– Chinese-style

dinner

• 2 Parameters– # of participants– Predefined vs.

Random Sequence

A

Willy

Keng-hao

17

Experimental ResultsScenarios Event Statistics Results

Dining Scenario

s

# Users

Activity Sequenc

e

Time Durati

on(secon

ds)

# of Dietar

y Behavi

or

Average Activity Interval

Behavior Recogniti

on Accuracy

#1 Afternoo

n tea

1 Predefined

73 12 6.08 100%

#2 Afternoo

n tea

2 Predefined

162 24 6.75 100%

#3 Afternoo

n tea

2 Random 913 78 11.71 79.49%

#4 Chinese-

style dinner

3 Random 1811 162 11.18 85.8%

18

Predefined Activity Sequence

Afternoon Tea (Single User)1. cut a piece of cake and

transfer it to the personal plate;

2. pour tea from the tea pot to the personal cup;

3. add milk to the personal cup from the creamer;

4. eat the piece of cake from the personal plate;

5. drink tea from the personal cup;

6. add sugar to the personal cup from the sugar jar.

Afternoon Tea (Multi-users)1. A cuts cake and transfers it to

A’s personal plate;2. B pours tea from the tea pot to

B’s personal cup; 3. A pours tea to A’s personal cup

while B cuts a piece of cake and transfers it to B’s personal plate;

4. A adds sugar from the sugar jar to A’s personal cup while B adds milk from the creamer to B’s personal up;

5. A eats cake and B drinks tea; 6. B eats cake from B’s personal

plate while A drinks tea from A’s personal cup;

7. A pours tea from the tea pot to both A’s and B’s personal cups.

19

Activity Recognition Accuracy in Scenario

#3

Dietary Behavior

# of Actual Events

Recognition Accuracy

Transfer event

41 70.73%

Eat event 37 89.19%

20

Causes of Misses in Scenario #3

Causes of misses # of misses of transfer

events

# of misses of eat events

Total

Event interference within the weighing

cell’s weight stabilization time

6 2 8

Weight matching threshold

2 0 2

Slow RFID sample rate

3 0 3

Touching table 1 2 3

Total of misses 12 4 16

21

Dietary Behavior

# of time

s

Recognition Accuracy

Weight Accuracy

Transfer dish A events

19 73.68% 68.42%

Transfer dish B events

29 79.31% 78.75%

Transfer dish C events

23 82.61% 79.19%

Transfer rice events

12 83.33% 81.88%

Transfer soup events

19 84.21% 80.16%

Eat events 60 88.33% 91.23%

Activity Recognition Accuracy in Scenario

#4

22

Causes of Misses in Scenario #4

Causes of misses # of misses of transfer events

# of misses of

eat events

Total

Segmented weight-change events

5 0 5

Eating before transferring food to personal containers

5 5 10

Weight matching ambiguity

7 0 7

Touching table 3 2 5

Slow RFID sample rate 3 0 3

Total of misses 23 7 30

23

Conclusion

• It is a smart object and a smart surface• It supports natural user interface• It supports fine-grained dietary tracking

at individual level• It is about human-centric computing• Accuracy can be improved further

24

Future Work

• Improving recognition accuracy• Removing constraints (assumptions)• Persuasive computing

– Encourage balanced diet– Encourage proper amount of diet

25

Questions & Answers

Thank You