A Cloud-Based Infrastructure for

Caloric Intake Estimation from

Pre-Meal Videos & Post-Meal Plate Waste Pictures

Vladimir Kulyukin Vikas Reddy Sudini

Department of Computer ScienceUtah State University

Heidi Wengreen Jennifer Day

Department of Nutrition, Dietetics, & Food SciencesUtah State University

Outline

● Background● Human-Centered Caloric Estimation

Infrastructure● Pilot Deployment● Discussion

Background

Motivation● Accurate caloric intake estimation is an open

research problem in health informatics, dietetics, and nutrition management

● Obesity, diabetes, blockage of coronary arteries are related to mismanaged diets

● There is a growing need for robust methods for accurate and periodic caloric intake estimation

Vision-Based Approaches● Many current R&D efforts approach the problem of caloric

intake estimation through complete automation by means of computer vision [1, 2]

● Vision-based approaches estimate caloric intake from static pictures and, increasingly, videos

● Vision-based approaches exhibit high percentages on selected food items [3] but underperform due to low image quality, absence of reliable ground truth baselines, and highly variable food textures

Disengagement of the Nutritionist

● One reason, relatively unexplored in the literature, why completely automated solutions may underperform is that they do not integrate nutritionists into the process

● Nutritionists are typically asked for feedback post-factum when the ground truth is needed to estimate the performance of a given algorithm on a given set of images

● Some nutritionists disengage because they may feel that they have little or no stake in the system

PNUTS: Persuasive NUTtrition Mangement System

● To make nutritional data collection and estimation more manageable and enjoyable both to end users and nutritionists, we have been developing PNUTS

● PNUTS seeks to shift research, training, and clinical practices in nutrition management toward persuasion and community-oriented, context-sensitive nutrition decision support

● PNUTS is inspired by the Fogg Behavior Model [4]: motivation alone is insufficient; triggers and abilities are fundamental

Cloud-Based Infrastructure for

Human-Centered Caloric Intake Estimation

Proposed Infrastructure● An objective of this paper is to propose a nutrition data

collection and analysis infrastructure ● The infrastructure enables clients to submit pre-meal videos

and post-meal plate waste pictures● The infrastructure recognizes the critical role of human

nutritionists and aims to keep them integrated into the process

● The infrastructure is designed to educate new generations of nutritionists

Home Page of PNUTS

Three Main Systemic Roles

● The system has three main systemic roles:ClientNutritionistProgram Coordinator

● There is also the system administrator but this role is outside the scope of this paper

Client● The client is any user interested in accurate caloric intake

estimations of consumed foods and beverages● The client uses a web-enabled device to register with the

system● At breakfast, lunch, and dinner the client takes a short pre-

meal video and does an optional voice recording of the foods and beverages to be consumed

● At the end of the meal the client takes a still picture of plate waste

Client Meal Data Upload

Client's Pre-Meal Video

Client's Post-Meal Plate Waste

Client's Energy Intake Estimation Request (EIER)

● When the client submits a pre-meal video, post-meal static plate waste picture, these data, together with the client's ID and a time stamp, constitute an energy intake estimation request (EIER)

● Each EIER is saved in an SQL database and viewed subsequently by the nutritionist

Nutritionist

● The nutritionist also registers with the system to review and evaluate submitted EIERs

● The nutritionist watches the videos and looks at plate waste pictures

● Caloric intake estimates are based on USDA's National Nutrient Database (ndb.nal.usda.gov/ndb)

Pending EIERs for Nutritionists

Five States of EIERs● An EIER is in one of the five states: unprocessed, pending,

processed, conflicting, and resolved

● Unprocessed – received by the system but not evaluated by any registered nutritionist

● Pending – evaluated by one nutritionist

● Processed – evaluated by two nutritionists without a conflict● Conflicting – evaluated by three nutritionists whose evaluations

disagree by more than 10%● Resolved – conflicting request evaluated by a program coordinator

Example of Two Conflicting EIERs

EIE Matching: Case I

EIE Matching: Case 2

Program Coordinator

● The Program Coordinator is also a registered nutritionist who has the authority to resolve EIER conflicts

● In principle, the system is designed to have multiple program coordinators, each supervising a number of nutritionists

● When a conflicting EIER is detected by the system, an appropriate program coordinator receives an email

● When the program coordinator logs in, the program coordinator sees a list of conflicting EIERs and resolves the conflict

List of Conflicting EIERs

Implementation Details

● The proposed infrastructure has been implemented in Java using Java Servlets, Java Server Pages (JSPs), and JBOSS

● The current cluster has two nodes: one master and one slave

● All databases are implemented with MySQL

Pilot Deployment

Student Training

● The pilot version of the infrastructure was deployed and integrated into NDFS 4750: Transition to Professional Practice taught at Utah State University in the spring 2015 semester

● All students enrolled in NDFS 4750 received caloric estimation training prior to acting as clients and nutritionists

● Each student watched and evaluated 14 training pre-meal and post-meal video pairs and was required to complete by hand a caloric intake estimation form

● Some students had to undergo additional training

Sample Caloric Intake Estimation Form

Data Collection

● 28 students, acting as clients, provided EIERs for digital data of 61 eating occasions

● Each client completed a paper-pencil three day food record of all foods eaten and submitted to the system

● 28 students, acting as nutritionists, provided energy intake estimates for these EIERs

● 407 pre-meal videos, 407 post-meal pictures were collected

Sample 3-Day Food Record

Preliminary Analysis

● The mean estimate of the 28 nutritionists was 383 kcalories (range: 35 – 936)

● Agreement among the nutritionists was examined with intra-class correlation coefficients (ICCs): ICC = 0.89, p<0.001

● Agreement was lower for digital data from eating occasions that included a single food or five or more food items: ICC=0.78, 0.77, respectively; p-value=0.023 and p-value=0.002, respectively

● These values are at least as high as the ICCs reported with similar studies in cafeteria settings and free living adults [5, 6]

Discussion

Observations

● The pilot version of the infrastructure functioned flawlessly during the entire semester

● There were two main complaints:

The clients complained that the system did not allow the to submit partial caloric estimations with an op-tion to complete them later

The nutritionists complained about the necessity to use the external database at ndb.nal.usda.gov and retype the information into a separate form

Observations● In an informal qualitative survey conducted with the students they

said that the system made them feel as part of a community of users

● Many students said that they would use the system as registered nutritionists after they graduate

● When asked about the data entry difficulties, the students said that these difficulties are compensated by not having to meet with the clients in person

● The system has turned out to be a valuable training tool for undergraduate nutritionist students

Acknowledgments

● We are grateful to Dr. Sheryl Aguilar for letting us pilot the system in NDFS 4750 she taught at USU in the spring 2015 semester

● We express our gratitude to all students enrolled in NDFS 4750 for testing the system as clients and nutritionists for their effort and feedback

Selected References

[1] Bosch M., Zhu F., Khanna N., Boushey C.J., and Delp E. “Combining global and local features for food identification in dietary assessment.” IEEE transactions on Image Processing . 2011:1789-1792. doi:10.1109/ICIP.2011.6115809.

[2] Martin, C.K., Han, H., Coulon, S.M., Allen, H.R., Champagne, C.M., and Anton, S.D. (2009). “A novel method to remotely measure food intake of free-living people in real-time: The Remote Food Photography Method (RFPM).” British Journal of Nutrition, 101, 446-456. PMCID: PMC2626133.

[3] Chen, M.Y., Yang,Y., Chia-Ju Ho, C., Wang, S., Liu, S., Chang, E., Yeh, C., & Ouhyoung. M. “Automatic Chinese food identification and quantity estimation.” In Proceedings of SIGGRAPH Asia Technical Briefs (SA '12). ACM, New York, NY, USA, , Article 29 , 4 pages, 2012. DOI=10.1145/2407746.2407775.

[4] Fogg, B.J. “A behavior model for persuasive design.” In Proceedings of the 4th International Conference on Persuasive Technology. Arctile 40. ACM, New York, USA, 2009.

[5] Wengreen, H.J., Madden G.J., Aguilar S.S., Smits R.R., Jones B.A. “Incentivizing children’s fruit and vegetable consumption : results of a United States pilot study of the Food Dudes program.” Journal of Nutrition Education Behavior. 2013;45(1):54-9.

[6] Martin C.K., Han H., Coulon S.M., Allen H.R., Champagne C.M., Anton S.D. “A novel method to remotely measure food intake of free-living individuals in real time: the remote food photography method.” British Journal of Nutrition. 2009;101:446-456.