How To Monitor Human Behaviors: Current Status & Our Approaches

October 24th, 2013

Pil Ho Kim, pilho.kim@unitn.it University of Trento, Italy

1

Current Status

• What did you feel when you read the title of this talk?

• Then let’s see what is going on now. We will review three software applications:

• Google Now

• Moves

• Our work

Google Now• Objective: An Intelligent Personal Assistant (Know-it-all)

• http://www.google.com/landing/now/

Images © Google

Main Services (as of now)

4

#1. Manage your day

#2. Stay connected

#3. Be a local

Images © Google

Information Sources

• Gmail Cards (GC)

• Web History (WH)

• Synched Calendar (SC)

• Current Location (CL)

• Google Plus (G+)

• User Reminder (UR)

• Current Time (CT)

• “Remind me” Button (RB)

• Google Offer Service (GO)

• User Selection (US)

• Event Booking (EB)

• Google Finance Portfolio (GF)

• Smart TV (ST)

• Collaboration with 40 more companies (http://www.google.com/landing/now/integrations.html)

#1. Manage Your Day

• Boarding pass (GC)

• Activity summary (CL)

• Next appointment (SC, CL)

• Weather (CL)

• Traffic & Transit (CL, WH)

• Flights (GC)

• Hotels (GC)

• Restaurant reservations (GC, CL)

• Events (GC, RB)

• Packages (GC)

• Friends’ birthday (G+)

• Your birthday (G+)

• Time reminders (UR, CT)

• Location reminders (UR, CT, CL)

• Event reminders (RB)

• Saved offers (GO)

• Car rentals (GC)

Sources: Gmail Cards (GC), Web History (WH), Synched Calendar (SC), Current Location (CL), Google Plus (G+), User Reminder (UR), Current Time (CT), “Remind me” button (RB), Google Offer service (GO)

#2. Stay Connected

• Zillow (Search nearby real estate listings) (WH, CL)

• Sports (US)

• Movies (WH, CL)

• Fandango (Movie booking) (EB)

• Concerts (WH, CL)

• Stocks (WH, GF)

• Public alerts (WH, CL)

• Developing story & breaking news (WH)

• Research topic (WH)

• New albums (WH)

• New books (WH)

• New video games (WH)

• New TV episodes (WH)

• TV cards (ST)

• Concert ticket (GC)

Sources: Gmail Cards (GC), Web History (WH), Current Location (CL), Google Plus (G+), User Selection (US), Event Booking (EB), Google Finance Portfolio (GF), Smart TV (ST)

#3. Be A Local

• Public transit (CL)

• Places (CL)

• Nearby attractions (CL)

• Nearby events (CL)

• Nearby photo spots (CL)

• Translation (CL)

• Currency (CL)

• Time at home (CL)

Sources: Current Location (CL)

Supported Platforms

Desktop supports are under way through Chrome notification:source: https://support.google.com/mail/answer/1075549?hl=en

Images © Google

User Interfacing•Majority of natural language interfaces through

texts and also speech.

•Cards, cards and cards (Unified representation of user’s personal events).

• Sources from the user through interactions and monitoring in consideration of contexts from outer sources.

•Notification methods: App and widget for mobile and notification bar for desktop

What Others Think on Google Now?

Google Now for iOS review: straddling the creepy line (May 2nd, 2013), The Telegraph

•As more services are added, it will keep track of more of your choices. These things are either creepy or cool, depending on your definition of privacy.

•Unusually for Google it’s a beautiful, graceful interface that can present a lot of information in a concise, aesthetically pleasing format.

• Features can be turned off with impressive granularity. Also the battery life is issued.

Source: http://www.telegraph.co.uk/technology/mobile-app-reviews/10032788/Google-Now-for-iOS-review-straddling-the-creepy-line.html

What Others Think on Google Now?

Review: Google Now useful as a supplement to Siri (4 May, 2013), USA Today

• Google Now’s power lies in giving you information you need to know before you have to ask. It works best as a supplement to Siri, rather than a replacement.

• Google Now's judgments on what information I need to know can be sketchy at times.

• Despite getting stray information at times, I find Google Now useful enough to leave it on. I could always customize the service by telling it never to give me stock quotes, for instance.

Source: http://www.usatoday.com/story/tech/2013/05/04/google-now-siri/2129755/

What Others Think on Google Now?

Google’s Virtual Assistant Google Now Is Awesome - Here’s Why I Never Want To Use It (22 June, 2013), Business Insider

• Google Now puts Siri to shame – it's more accurate and far more versatile.

• Trouble question:  It begs to be relied upon and integrated into your lifestyle in a major way. Even a casual Google Now user is still feeding all his data to it every time he uses the web.

• The intrinsic nature of Google Now is that it's designed to be blindly relied upon, to be a "life aid." What happens when it goes kaput for some reason? All those things that you were used to not having to think about become problematic.

• I'm in charge of my phone. My phone's not in charge of itself. Google Now nearly represents the phone coming to life, acting on its own and without your direction. It becomes slightly less your phone and you become a slightly less mindful.

Source: http://www.businessinsider.com/the-case-against-google-now-2013-6

What Others Think on Google Now?

Google Now could be the injection of intelligence smart watches need (16 Oct, 2013), theguardian

• Google is reckoned to be working on a Google Now-focused smart watch codenamed ‘Gem’ which could finally make smart watches intelligent (Confirmed yesterday -- Source #2).

• Industry experts reckon none of the current smart watches has truly "smart" functionality. And that leaves the door wide open for something that multiple sources are murmuring about: a Google Now-powered wearable.

• Google started a project (Codename “Gem”) to bring Google Now to the smart watch. FYI, Google Glass already supports Google Now.

Source #1: http://www.theguardian.com/technology/2013/oct/16/google-now-gem-smartwatchSource #2: http://online.wsj.com/news/articles/SB10001424052702304655104579165080029933904

Development History• Late 2011: Enhancing existing Google Voice Search

(Codename: Majel)

• June 27, 2012: Google Now appeared in Android 4.1 Jelly Bean Google I/O

• October 29, 2012: UI changed to Cards with information pulled from Gmail account, user location, search history and more have been added since then.

• March 21, 2013: iOS version of Google Now

• May 15, 2013: Google I/O 2013 announced the upcoming release of Google Now on desktop platforms; the feature will be accessible only via Google Chrome or Google Chrome OS.

Trends on “Google Now”

Google Now comes to iPhone

Images © Google Trend

Today Google Now Told Me..

Personal Review (I)

•Great example in merging heterogenous sources to provide intelligent & personalized services.

• Shows how Google monitors personal behaviors and what it classifies.

• The concept of Google Card adds strong flexibility in extending personalized services.

Personal Review (II)

• It looks clear that Google runs at least multiple instances of real-time listeners (CEP) for each Google Now user.

• Looks promising but not working well for me. Accuracy (in multiple aspects) could be an issue.

This email was exchanged on Mar 2013. The content is in fact a customer service email asking

about my old flight travel (11/10/2012 - 24/10/2012), not

today (24/10/2013).

Moves• Tracks the amount of exercise -- activity tracking app.

• http://www.moves-app.com

Features

Images © Moves-app.com

Our Approaches

+

Google Now Moves

And keep you data under your hood !Images © Google & Moves-app.com

Big Picture

27

Data Semantic

Big Picture

27

Sources Event Processor KnowledgeData Semantic

Events & System

• We need to develop a system for event detection.

• We mean an event as a change from multiple sources.

• Personal events are so diverse and non-static.

• So we do not define an event.

• But our system provides a way for users (or developers) to define events of their own with good enough freedom.

• Event streams can be modeled as a pattern to build up a personalized knowledge base (i.e. an ontology linking a word or words in semantic relations to low-level changes).

Implementation Strategies

• The amount and quality of information sources directly affects the machine intelligence.

• The challenges are then (1) a backend to collect enormous data from various sources and (2) methods to mine big data.

• We have currently implemented (1), which will get better, and are working on (2).

Mining Data to Events

33

• Many methods exist like signal processing, statistical analysis and probabilistic state modeling methods.

• Why Complex Event Processing (CEP)?

• Rich supports for temporal logics and complex controls.

• Encapsulate above methods as event sources using, what are called, Platform Independent Model (PIM) and Computational Independent Model (CIM) that are under various testing for implementation.

• Enable continuous learning rather than offline learning. Though, it should be noted that real-time pattern modeling needs significant efforts.

Complex Event Processing

34

CEP is a technology to help you process complex events. So don’t be scared but start using it as a tool.

Video © Opher Etzion, IBM Research

Source: http://www.slideshare.net/opher.etzion/event-semantics-and-model-presented-at-the-event-based-multimedia-workshop-presentation-at-acm-

multimedia-2013

Information Sources (as of now)

• Accelerometer (AC)

• Ambient Temperature (AT)

• Battery Status (BS)

• Light Level (LL)

• Gravity (GV)

• Gyroscope (GS)

• Linear Acceleration (LA)

• Orientation (OE)

• Magnetic Field (MF)

• Rotation Vector (RV)

• GPS Location (GL)

• Network Location (NL)

• WIFI (WF)

• Pressure (PL)

• Relative Humidity (RH)

We need more sources to be intelligent.

How to Detect Events?

36

Method #1. User Records

How to Detect Events?

37

Method #2. Manual Data Analysis

How to Detect Events?

38

Method #3. Automated Event Detection Using CEP

Esper?http://www.espertech.com

39

• An esper refers to an individual capable of telepathy and other similar paranormal abilities.

• Sorry.

• Esper is an open source event stream processing (ESP) and event correlation engine (CEP).

• Esper is an extensible middleware supporting:

• Java, .Net, Map or XML events

• Runtime statement management

• API or configuration driven

• Plug-in SDK for functions, aggregations, views and pattern detection extensions

• Agile development and testing, integrates with Eclipse and JUnit

• Adapters: Adapters to any possible transport exist through integration with Apache Camel, ServiceMix and Mule ESB. EsperIO and Enterprise Edition provide additional adapters.

Esper Example (First Try)

40

Tracking user’s indoor &

outdoor location

Esper Example (First Try)

40

Sensor-specific source selection

Stream dependent

domain knowledge

Too complex logic in the query statement to output an event detection status

Tracking user’s indoor &

outdoor location

Current Supports:Now You Can Write Much Less

41

Current Supports:Now You Can Write Much Less

41

An abstract event class that encapsulates heterogenous sources.

An abstract event class that encapsulate heterogenous sources to trigger next actions (ex. pushing/pull notification, auto recording, pattern mining, etc.)

Generalized status retrieval

State changes are now modeled as a pattern.

Domain knowledge becomes independent functions.

Thank You.

43

Any question?

References

44

Please refer following references for details.

• Opher Etzion, Modeling and semantics of events and contexts, http://www.slideshare.net/opher.etzion/event-semantics-and-model-presented-at-the-event-based-multimedia-workshop-presentation-at-acm-multimedia-2013

• Google Now: http://www.google.com/landing/now/

• Moves: http://www.moves-app.com