Emerging Topics on Personalized and Localized Multimedia ...

1Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann

Emerging Topics on Personalized and LocalizedMultimedia Information Systems

November 3, 2014

Yi Yu, Roger Zimmermann

School of Computing, National University of Singapore

-- Exploring Interesting Aspects Hidden in Location-Aware Multimedia Data from Individual Level to Society Level --


Personalized Multimedia Service

S. Ramanathan, P. V. RanganIEEE MultiMedia archive Volume 1 Issue 1, March 1994 Page 37-46

Content Analysis

User

Profile Construction

User’s interests

Relevance feedback

Content Selection


Society-based User behaviors

User1, .…. , Usern

Profile Construction

User interests

Relevance feedback

Personalized Recommend.

Individual User

Participatory Sensing

Location Information

Multimedia Data

Aggregating data of more users

Evolved Framework of Personalized Multimedia Service


People Life & Mobile Technologies


Smart Mobile Services

Understanding a user, his physical and mental state


Smart Search


https://ginger.io/

Smart Well-Being


Smart Watch


People with Data


Data about People (1)


Data about People (2)


What Will be Covered ?

Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencingWeb-based map personalizationParticipatory Sensing of Venue via Multimedia Events


Correlating between Preference-Aware Activities

From a user-centric point of view Extract activity data

• E.g., online presence, physical check-ins Category by semantic concepts


Smartphone

appVideo/Audio Association

UGV with Soundtrack

Ready for Sharing

Geo-feat., GVisual feat., F

Raw UGV

SVMhmm

Training

Scene MoodRecognition

f (MG,G), f (MF, F) SoundtrackRecommendation

Training dataset with Geo-tagged video

External Data Source

MusicE.g.: Mp3 Files, Tags

Server side

Personalized Music Soundtrack

(1)

(2)

(5)

Rajiv Ratn Shah, Yi Yu, Roger Zimmermann, ACM MM’14

Personalized Video Soundtrack Recommendation

Moodannotation

Visual feat., FGeo-feat., G

ModelsMG, MF

mood

Listening history

Personalization

offline Online

(3)

(4)

http://www.comp.nus.edu.sg/~yuy/matsrc_build_Updated.zipCodes at


Geo-sensor info recorded with video Geo-sensor info geo-category geo feature mood tags with geo-aware likelihood CG

Visual content Visual feature mood tags with visual-aware likelihood CF

Likelihoods of mood tags (CG and CF) Mood tags with large likelihoods scene mood C

Mood Recognition from Video

Geo feature G

Visual feature F

Late

fusio

n

SVMf (MF, F)

SVMf (MG,G) CG

CF

Scene mood C

Geosensor

info

Visual content

Geocategory

Visual feature extraction(e.g., color histogram)

APIBag ofword

Rajiv, Ratn Shah, Yi Yu, Roger Zimmermann, ACM MM’14


Matching Songs with User Preference

Sweet

Funny

Sad

Songs

Songs

Songs

Music databaseWith mood-tag as index

Scene mood C

Soundtrackretrieval

Listeninghistory

Personalizationby correlating audio features

Initial song list User specific songs

Mood tagSongs

Rajiv, Ratn Shah, Yi Yu, Roger Zimmermann, ACM MM’14



Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencing Personalization of web-based map generalizationParticipatory Sensing of Venue via Multimedia Events


User Preference Profiling: Background

User modelingAn understanding of a user ( characteristics,

preferences, needs)

Modeling user location historyProvide personalized servicesGeographic data semantic geo-category

• (e.g., coordinates bar, shopping mall)


Location-aware user profiling (approaches)Term Frequency-Inverse Document Frequency (TF-IDF)Sparse Additive Generative model (SAGE)Latent Dirichlet Allocation (LDA)

Examples Social check-ins & location preferenceCheck-in patterns & shopping habitTopical diversity, geographical diversity interest diversity

• Tweet prediction• Location inference

User Preference Profiling : Background


Jie Bao, et al ACM GIS’12Location history (document), categories (terms) User preference hierarchy (TF-IDF)

Category Name Number of sub-categories

Arts & Entertainment 17

College & University 23

Food 78

Great Outdoors 28

Home, Work, Other 15

Nightlife Spot 20

Shop 45

Travel Spot 14

Check-ins & Preference

Number of venue c’ being visited by u

Number of users visiting venue c’

w1×sim1(ua,ub)+w2×sim2(ua,ub)

Total similarity between a and b

'| { . : . '} | | |. log

| . | |{ : ' . } |cu v v c c Ru w

uV u c u C== ×

∈


Top 9 categories or 410 sub-categoriesTrade area analysisNot effective or too high dimensional

User profilingHistogram of user check-ins (sub-categories)Latent Dirichelet Allocation (LDA)

• Assumption– document: a mixture of topics– Topic: probability of mentioning a word

• Goal: calculate proportion of documents by examining word distribution

Check-ins & Habit Pattern: Background

David M. Blei, et al JMLR’03


Yan Qu, et al WWW’13

Check-ins & Habit Pattern Distribution of topics (users) User (document) Topic (term)

Store profile: histogram of topics Generated by all customers


For a term in a model Term frequency is Log-frequency is Get distribution via normalization Addition of several models

exp( )( | )exp( )

v v

v vv v

p v β φφβ φ

= =

++++=++

v

gv

uvv

gv

uvvguvp

)exp()exp()|( 0

00

φφφφφφφφφ

gv

uvvv ββββ ⋅⋅= 0

: Basic reference model: Difference between one model and the reference model

: Difference between another model and the reference model

Termfrequency

Change rate of term frequency

Change rate of term frequency

Preliminaries in Log Space

vβvv βφ log=

)( 00 βφ)( uu βφ)( gg βφ


SAGE models the difference from a background distribution in log-space Use to denote the background model Other components , used to model the differences from the background

model Sparsity-inducing for each specific model (difference of term frequency) Generative facets combination through simple addition in log space

Jacob Eisenstein, ICML’11

Sparse Additive Generative Model

0φ

0φ

uφ

gφ

Background

Topic

Perspective

+

SparsityOnly a few non-zero items

Term distribution

uφ gφ


A probabilistic model considers User locations, global topics, regional language models

SAGE for Tweet PredictionLiangjie Hong, et al WWW’12

Pick a region r

Pick a topic z

Generate tweet d

Background LM

Topic Region LM

: user dep. distr.over regions

: global distr. over regions

: user dependent distribution over topics

: global distribution over topics

: regional distribution over topics

: global distribution over terms

: region-dependence of terms: topic matrix, each row is a distribution over terms


Location Prediction for New TweetBased on the words in the tweet and user informationMost proper region given

Pick a region r

Pick a topic z

New tweet

Find

regi

on

Topics

User info

Maximize likelihood

likelihood

r: user dep. distr.over regions

: global distribution over topics

: user dependent distribution over topics

: global distr. over regions

: regional distribution over topics



Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencing Web-based map personalizationParticipatory Sensing of Venue via Multimedia Events


A virtual perimeterCoverage of a radio cell or Wi-Fi access pointOr manually specified geographic shapeDifferent shapes (e.g., circles, rectangles, polygons)

Basic ideaUsers enter or exit boundaries of areas send notification

Geo-Fence


Personalized Geo-Fencing (1)

Location-aware social networks


Personalized Geo-Fencing (2)Keep track of children


Personalized Geo-Fencing (3)Home securityAwareness what is happening at your property


Pairing points with polygons Scalability

• Big-data

Basic spatial predicatesINSIDE, WITHIN

A novel geo-fencing algorithmSimple but effective and efficientLSH + probing

Efficient Geo-Fencing Algorithm

Yi Yu, SuhuaTang, Roger Zimmermann, ACM SIGSPATIAL GIS’13


Geo-Fencing: Points & PolygonsPoints Multiple instances

• A unique sequence number Moving

Polygons Multiple instances

• A unique sequence number Changing

Sequence numbers Same space & no overlapping A timestamp

sequence


Geo-Fencing: Polygon Instances Two types With a single out-ring With a single out-ring & multiple inner-rings

-1.5 -1 -0.5 0 0.5 1 1.5

x 104

-1.5

-1

-0.5

0

0.5

1

1.5x 10

4ID=7, seq=[7,410293,419522,701191,734903]

1

2

3

4

5

Tips: Inner rings

• Inside outer-ring• Separated from

outer-ring


-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3x 10

4

-1

-0.5

0

0.5

1

1.5

2

2.5

3x 10

4

1

2

34

5

6

789

10

11

12

13

14

15

X axis (m)

Y a

xis (

m)

Geo-Fencing: Polygons, Point, Edges

# edges of 15 polygons (200)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

285 255 235 196 264 250 240 239 226 226 242 153, 15

152, 20

250 217


Geo-Fencing: INSIDECrossing number algorithm

Inside (number of intersections = odd) Checking each edge

Exploiting minimum bounding rectangle (point outside MBR is surely outside polygon)

LSH-based acceleration (point inside MBR)

MBR: minimum bounding rectangle

V1,9

V1,8

V1,11

V1,10

V1,7 V1,6

V1,4

V1,3

V1,2

V1,1

P V2,1V2,2

V2,3V2,4

V2,5

V1,5


Geo-Fencing: WINTH

V1,9V1,8

V1,11

V1,10

V1,7 V1,6

V1,5

V1,4

V1,3

V1,2

V1,1

P

V2,1V2,2

V2,3V2,4

V2,5

dthdth

Point outside MBR but within a distance A rectangle centered at the point, edge length=2 * dth

• Non-overlap, point surely not WITHIN a distance


Point file

Polygonfile

Polygon cache

MBRsin R-tree

Edges in LSH table

R-tree based pre-filtering

LSH-based INSIDE

detection

LSH-basedWITHINdetectionPo

lygo

n m

anag

emen

t

INSIDE result

WITHIN result

Pairing engine

Geo-Fencing: Scalable Framework A point inside MBR of a polygon Adapt to crossing number algorithm A probing scheme to lookup edges



Geo-Fencing: Point, Latest Instances of Polygons

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3x 10

4

-1

-0.5

0

0.5

1

1.5

2

2.5

3x 10

4

1

2

34

5

6

789

10

11

12

13

14

15

X axis (m)

Y a

xis (

m)

15 MBRs, three groups (an R-tree) MBR: minimum bounding rectangle


A separate hash table for each polygon

An edge within a bucket (x range within a bucket’s x sub-range)

Hash function T = (Xmax – Xmin) / N HashKey(x) = int ((x – Xmin)/T)

An edge (x1, y1)—(x2, y2) stored in buckets from key1 to key2

key1=HashKey(x1) key2=HashKey(x2)

Geo-Fencing: INSIDE Detection

B0 B1 B2 B3 BN-1

P

x0 x1 x2 x3 x4 xN

Xmin

Xmax



Geo-Fencing: WITHIN Detection

x0 x1 x2 x3 x4 xN

Buckets probed for WITHIN dth of Polygon

B0 B1 B2 B3 BN-1

dth

dth

dth

dth

III

III IV

P3dth dthPT

PL

PB

PRP2

WITHIN in three cases Inside polygon

Inside inner ring

Outside outer ring

Optimization Range of a point Divide outer area into 4

ranges Only check edges in the same

range as the point

P1

P1

P2P3

P3

probed



What Will Be Covered ?

Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencing Web-based map personalizationParticipatory Sensing of Venue via Multimedia Events


Web-Based Map Personalization (1)Describing a part of the physical world for the user


Web-Based Map Personalization (2)

Map customization: annotating personal maps with e.g., landmarks, routes, custom shapes

Map simplification: removing non-relevant details while retaining personalized details on small screens


Map Customization

My Maps


Map Simplification: Background (1)

Producing maps with less detail

Personal GIS, business mapping applications

Reducing data without losing general shape of map

http://openstreetmap.us/~migurski/streets-and-routes/


Map Simplification: Background (2)

User preferences Specify his own constraining points to

control where to keep more details of his personal map

User 1 ( p1, p2, p3)

User 2 (p1, p3, p4, p5)


Map Simplification: TaskAccess to linear geometries along with points

Reducing #vertices while preserving topological constraints

-8.1 -8.05 -8 -7.95 -7.9 -7.85

x 106

5.2

5.25

5.3

5.35

5.4

5.45

5.5

5.55

5.6

5.65x 10

6

Inputs: linear geometries (27) & constraining points (26)


Topological relationships between original set of input linear geometries does not change after simplification

Relationship between constraining points and linear geometries before and after simplification does not change

Map Simplification: Two Constraints

http://mypages.iit.edu/~xzhang22/GISCUP2014/problem.php

(1) before

(1) after

(2) before

(2) after


Douglas-Peucker algorithm (1973)

Visvalingam-Whyatt algorithm (1993)

Map Simplification: Two Algorithms


First and last points to be kept Point having maximum

perpendicular distance P4 in Fig. 1 is larger than the

tolerance

Whole linear geometry split into two parts at P4 Apply to linear geometry P2 and P7 have the maximal

perpendicular distance

(2) Tentative simplified segment

(1) Initial simplified line

(3) Final version of simplified line

P6P5

P7

P8

P3

P4

P2

P1

P6

P5

P7

P8

P3

P4

P2

P1P4

P6

P5

P7

P8

P3P2

P1

Tolerance

Douglas-Peucker Algorithm & Example (1973)


Pseudo Codes for Douglas–Peucker Algorithm--using a given threshold tolerance--

function DouglasPeucker(PointList[], epsilon)// Find the point with the maximum distancedmax = 0, index = 1, end = length(PointList)for i = 2 to (end - 1) {

d = shortestDistanceToSegment(PointList[i], Line(PointList[1], PointList[end]))if ( d > dmax ) { index = i, dmax = d }

}if ( dmax > epsilon ) { // Recursive call

recResults1[] = DouglasPeucker(PointList[1...index], epsilon)recResults2[] = DouglasPeucker(PointList[index...end], epsilon)// Build the result list

ResultList[] = {recResults1[1...end-1] recResults2[1...end]}} else {

ResultList[] = {PointList[1], PointList[end]}}return ResultList[]


Visvalingam-Whyatt Algorithm & Example (1993)

(2) Effective area for P6

(1) Effective area for P2(minimal )

(3) Simplified line with six vertices

P6P5

P7

P8

P3

P4

P2

P1

P6

P5

P7

P8

P3

P4

P2

P1

Original line

The triangle formed by original point and its immediate neighbors

Computing effective areas for all original points

P6

P5

P7

P8

P3

P4

P2

P1

Iteration, effective areas of points adjacent to the removed one are recalculated


Pseudo Codes for ModifiedVisvalingam-Whyatt

ResultList = PointList.clone()for i=2 to i=length(ResultList)-1

Comp. effectiveArea[i] of ResultList[i], (triangle by ResultList[i-1, i, i+1])while length(ResultList) >number_to_keep

Find the point (minIndex) with least effective area// Remove the point with least effective areaif violate(PointList[minIdx], Constraint[])

effectiveArea[minIdx] = ∞else

update effectiveArea[minIdx-1], effectiveArea[minIdx+1]ResultList.remove(minIdx)

return ResultList

Iteratively remove the point with least effective areafunction VisvalingamWhyatt(PointList[], number_to_keep)


Simplification

Before After

Maintain the topological consistency & satisfy point constraint

-8.1 -8.05 -8 -7.95 -7.9 -7.85

x 106

5.2

5.25

5.3

5.35

5.4

5.45

5.5

5.55

5.6

5.65x 10

6

-8.1 -8.05 -8 -7.95 -7.9 -7.85

x 106

5.2

5.25

5.3

5.35

5.4

5.45

5.5

5.55

5.6

5.65x 10

6


(a) User A ’s constraining points (b) User B ’s constraining points

-8.1 -8.05 -8 -7.95 -7.9 -7.85

x 106

5.2

5.25

5.3

5.35

5.4

5.45

5.5

5.55

5.6

5.65x 10

6

-8.1 -8.05 -8 -7.95 -7.9 -7.85

x 106

5.2

5.25

5.3

5.35

5.4

5.45

5.5

5.55

5.6

5.65x 10

6

A

B

ExampleRealizing user preference by setting

constraining points



Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencingPersonalization of web-based map generalizationParticipatory Sensing of Venue via Multimedia Events


Participatory Sensing

Concept of communities

Using personal mobile devices and web services to systematically explore interesting aspects

E.g., urban computing, public health, cultural identification, mobile multimedia computing

http://en.wikipedia.org/wiki/Participatory_sensinghttp://www.mobilizingcs.org/about/participatory-sensing


Venue-Sensing Systems

E.g., Foursquare, Instagram


https://foursquare.com/v/the-metropolitan-museum-of-art/427c0500f964a52097211fe3/photos

Participatory Sensing of Venue via Multimedia Events


Sensed Data in FoursquareVast volumes E.g., check ins, venue photos, venue tips

Valuable knowledge


Associated with geographic category (e.g., beach, food)Leveraged for user activity analytics

Heterogeneous Information


Implications of Sensed Data in FoursquareUser physical activities & online sharing

behaviors Personalized information & participatory

sensingApplied to research Location-aware recommendation

• E.g., providing media advertisement, travel plan

Urban computing• E.g., providing sustainability and outlook of

urban environment, people life quality, city planning, social sciences


Check ins in FoursquareAn intersection of virtual social networks and

physical worldAs of December 2013, 45 million registered users

with 5 billion check-ins Spreading the world about their favorite spots

http://en.wikipedia.org/wiki/Foursquare


Business storefronts and interiors (e.g., restaurant) and service contents

Photos in Foursquare


Tips in Foursquare


#Venues #Users #Check-ins #Tips #Photos

LA 63,991 166,922 39,693,415 321,386 906,820

NYC 126,658 341,545 109,230,334 890,750 1,821,591

Empirical Observation of User Activities in Foursquare: Motivation

Yi Yu et al

Distribution and relationshipObserve interesting things behind data

Release data and source codes used in this study


Empirical Observation of User Activities in Foursquare: Data Description

Yi Yu et al

A Foursquare venue A physical location

• E.g., Union Square Park (Outdoors & Recreation--Park), Times Square (Outdoors & Recreation--Plaza), John F. Kennedy International Airport (Travel & Transport--Airport)

Two regions (NYC & Los Angeles) 10 primary categories

• Arts & Entertainment, College & University, Event, Food, Nightlife Spot, Outdoor & Recreation, Professional & other places, Residence, Shop & Service, and Travel & Transport

Release data and source codes used in this study


1 2 3 4 5 6 7 8 9 100

0.2

0.4

Category index

Hist

ogra

m o

f #tip

NYCLA

1 2 3 4 5 6 7 8 9 100

0.2

0.4

Category index

Hist

ogra

m o

f #ph

oto

NYCLA

1 2 3 4 5 6 7 8 9 100

0.2

0.4

Category index

Hist

ogra

m o

f #ch

ecki

n

NYCLA

Distributions of tips, photos, and check-ins

1. Arts & Entertainment

2. College & University

3. Event

4. Food

5. Nightlife Spot

6. Outdoor & Recreation

7. Professional & other places

8. Residence

9. Shop & Service

10. Travel & Transport

newly added

Tow similarities: Similar in different regions

Similar trend in different categories


Sun Mon Tue Wed Thu Fri Sat10

4

105

106

Day of week

Hist

ogra

m o

f #ph

oto

FoodShop & ServiceNightlife SpotProfessional & Other PlacesTravel & TransportArts & EntertainmentOutdoors & Recreation

Sun Mon Tue Wed Thu Fri Sat10

3

104

105

Day of week

Hist

ogra

m o

f #tip

FoodShop & ServiceNightlife SpotProfessional & Other PlacesTravel & TransportArts & EntertainmentOutdoors & Recreation

Dynamics of Sharing Activities in Foursquare

Similar trend: more popular at weekends

More professional activities on weekdays

Some differences

More tips on weekdays

More photos on weekend


100

101

102

103

104

10-5

100

(a) #tip per venue

NYCLA

100

101

102

103

104

10-5

100

(b) #photo per venue

CCD

F of

#ve

nue

NYCLA

100

101

102

103

104

10-2

100

(c) #checkin per venue

NYCLA

CCDF for #tips, #photos, and #checkinsat Venues in Foursquare

Almost half venues have only one tip while few venues have more than 100 tips

A common trend: only few venues have a large number of events

CCDF: Complementary cumulative distribution functions


100 10

1 102 10

3

100

101

10210

3

10-4

10-3

10-2

10-1

#Tip per user#Photo per user

Hist

ogra

m

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Histogram of users in terms of <#tip, #photo>

100 10

1 102 10

3 104

10010

110210

3104

10-4

10-3

10-2

10-1

#Tip per venue#Photo per venue

Hist

ogra

m

0

0.02

0.04

0.06

0.08

0.1

0.12

Histogram of venues in terms of <#tip, #photo>

Different preferences: posting tips & photos


10-3

10-2

10-1

100

101

102

10-4

10-3

10-2

10-1

100

Inter-visit time (day)

CCD

F of

inte

r-vi

sit ti

me

TipPhoto

100

101

102

10-4

10-3

10-2

10-1

100

Inter-visit distance (km)CC

DF

of in

ter-

visit

dist

ance

TipPhoto

Distribution of inter-visit time and inter-visit distance

Inter-visit time: time interval between two successive events 50%: tips, 7.3 days, photos, 1.83 days; average: tips, 50.0days, photos, 17.3days

Inter-visit distance: distance between two venues successively visited) 50%: tips, 3.72km, photos, 4.03km; average: tips, 6.10km, photos, 6.67km

50%


Different interests over different categories Variations of tips or photos per-category

A non-uniform distribution (a large fraction of visits in few categories)

Measuring user interest

Interest Focus/Entropy (1)


Interest focus F: highest fraction of visits per category A high interest focus interest limited to a specific category

Interest entropy H: standard entropy Fraction of visits to a category as a probability Higher values (more uniform)


max iki k

ikk

vFv

=

c1 c2 c3 c4 cN

PDF

−=k

ikiki ppH 2logik

ikikk

vpv

=

Probability density function


Interest focus Nearly 50% users have an interest focus greater than 0.5 Many users have a primary interest

Interest entropy Only 20% users have an interest entropy of photo greater than 1bit The interest entropy of tips is lower


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

Focus per user

CCD

F of

#us

er

TipPhoto

0 0.5 1 1.5 2 2.5 30

0.2

0.4

0.6

0.8

1

Entropy per user (bit)

CCD

F of

#us

er

TipPhoto

CCDF: Complementary cumulative distribution functions

0.5


Initial Observations in Foursquare

Category dynamics among venue photo sharing, tip sharing, and check ins have analogous geo-temporal rhythms

Shared venue photos are highly relevant to food

Users prefer to share photos rather than tips

Venue photos are more important in promoting Venues


Discussion on This Study of FoursquareThe potential applications Location recommendation Trip planning Media advertising Urban environment improvement City operation

Distributing our data and source codes http://www.comp.nus.edu.sg/~yuy/MyAnalysis.zip

Collecting data from broader regions More detailed usage of sensed data Prediction models

Emerging Topics on Personalized and Localized Multimedia ...

Documents

Transcript of Emerging Topics on Personalized and Localized Multimedia ...