Affect in recommender systems
-
Upload
marko-tkalcic -
Category
Technology
-
view
509 -
download
2
description
Transcript of Affect in recommender systems
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Affect in recommender systems
Marko TkalčičUniversity of Ljubljana
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Presentation overview
I: LDOS presentation & Motivation II: What are emotions? III: Emotion in recsys – related work IV: Role of emotions in the MM consumption chain V: Affect in the decision-making stage Conclusions
Note: some material is not ours ... Fair use ...
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Part I: LDOS group at UL FE and underlying assumption
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS group at UL FE University of Ljubljana
– Faculty of electrical engineering• LDOS (Digital signal processing laboratory)
– Approx 15 members
Relevant people
Head: prof. Jurij Tasič
Andrej Košir
Marko Tkalčič
Ante Odić
Matevž Kunaver Tomaž Požrl
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS work on recommender systems 2002-2009: public movie datasets
– CBR– CF
2009-2012– Emotions– Context
2012 – – Decision making (affective + cognitive attributes)
• Ajzen model• Kahneman/Tversky model• ...
Basic RecSys
AffectiveRecSys
AffectiveComputing
Decision making Modeling in RecSys
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Underlying presumption of our work Recommender System = predictor of users‘ decision
making Decision making: EMOTIONS DO INFLUENCE
(c) Dilbert.com
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART II : What is affect/emotions/mood/personality
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART II : What are affect/emotions/mood/personality
Not well defined (wikipedia):
– Emotion = subjective, conscious experience
– Affect = experience of emotion (interchangable)
– Emotion vs. Mood:• Emotion = high arousal, short term• Mood = low arousal, long term
– Personality = accounts for the individual differences in the users’ emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]
Time (duration)
emotion mood personality
CHANGES FIXED
Psychophysiologicalexpressions
Biologicalreactions
Mental states
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Overview of emotions/moods Several definitions We take simple models, easy to incorporate in computers:
– Basic emotions– Dimensional model– Circumplex model
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Basic emotions Discrete classes model Different sets Charles Darwin: Expression of emotions in man and
animal Paul Ekman definition (6 + neutral):
– Happiness– Anger– Fear– Sadness– Disgust– Surprise
(c) Paul Ekman
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Basic emotions Discrete classes model Different sets Charles Darwin: Expression of emotions in man and
animal Paul Ekman definition (6 + neutral):
– Happiness– Anger– Fear– Sadness– Disgust– Surprise
(c) Paul Ekman
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Dimensional model Three dimensions
– Valence (positive vs. Negative)– Arousal (high vs. Low)– Dominance (power(less) over emotions)
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Dimensional model Three dimensions
– Valence– Arousal– Dominance
– (c) Lang, P. J. (1980)
Each emotive state is a point in the VAD space
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Circumplex model Maps basic emotions dimensional model (Posner et al.)
Arousal
Valence
high
negative positive
low
neutral
sadness
fear
disgust
surprise
joyanger
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
How to detect emotions Emotions are characterized:
– psychophysiological expressions, – biological reactions – mental states
SENSORS !!!
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
How to detect emotions? Explicit vs. Implicit Explicit
– Questionnaires (SAM)
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
How to detect emotions? Explicit vs. Implicit Explicit
– Questionnaires (SAM) Implicit:
– Work done in the affective computing community– Different modalities (sources):
• Facial actions (video)• Physiological signals ( GSR, EEG)• Voice• Posture• ...
– ML techniques• Classification (basic emotions)• Regression (dimensional model)
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Lie To Me Part1.avi (c) 20th Century Fox Main Character Cal Lightman = Paul Ekman
Defined the FACS(Facial Action Coding System)
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS Experiment 2 datasets:
– Posed (Cohn-Kanade dataset)– Spontaneous (LDOS-PerAff-1 dataset)
Input: Video streams of facial expressions as responses to visual stimuli
Output: emotive states as distinct classes
Gabor features kNN
Emotive state
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Results and conclusions Posed dataset: accuracy = 92 % Spontaneous dataset: accuracy = 62% Reasons for bad results:
– Weak learning supervision– Non optimal video acquisition (face rotation, occlusions,
changing lightning ...)– Non extreme facial expressions
Upcoming paper: IEEE Transactions on Multimedia
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Personality Definition:
User personality accounts for the individual differences in the users’ emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]
Ever-lasting Several models
Five Factor Model (FFM or Big5): Openness (inventive/curious vs. consistent/cautious) Conscientiousness (efficient/organized vs. easy-going/careless) Extraversion (outgoing/energetic vs. solitary/reserved) Agreeableness (friendly/compassionate vs. cold/unkind) Neuroticism (sensitive/nervous vs. secure/confident)
How to measure? Questionnaires:
International Personality Item Pool ( http://ipip.ori.org/ )
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS PerAff-1 dataset Emotive responses Ratings Personality data Videos of facial expressions 50 users, 70 items, sparsity=0 http://slavnik.fe.uni-lj.si/markot/Main/LDOS-PerAff-1
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART III : Related work on emotions in recsys
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART III : Related work on emotions in recsys
Emotions and personality Scattered work
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Gonzalez, 2007 ? Emotions as context in recsys?
Arapakis et al., 2009User affective feedback from automatic facial expression analysis
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Gonzalez, 2007 ? Emotions as context in recsys?
Arapakis, 2009User affective feedback from automatic facial expression analysis
Tkalčič et al., 2010 Affective user model
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Kaminskas, Ricci2011
find an appropriate musical score that would reinforce the affective state induced by the touristic attraction.
Gonzalez, 2007 ? Emotions as context in recsys?
Arapakis, 2009User affective feedback from automatic facial expression analysis
Tkalčič et al., 2010 Affective user model
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Kaminskas, Ricci2011
find an appropriate musical score that would reinforce the affective state induced by the touristic attraction.
Gonzalez, 2007 ? Emotions as context in recsys?
Arapakis, 2009User affective feedback from automatic facial expression analysis
Tkalčič et al., 2010 Affective user model
Lops et al, 2012 Ongoing work: emotion detection in the phase of presentation of the recommendations for generating unexpected and seredipitous recommendations
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Personality in recsys
Nunes et al., 2007 Personality as ???
Tkalčič et al., 2009Personality-based user similarity measure For the cold start problem
Rong Hu and Pearl Pu,
Personality-based user similarity measure For the cold start problem
Dennis and Masthoff,2012
Adapting persuasive (learning) Technologies to personality traits
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART IV: Emotions in the MM consumption chain
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART IV: Emotions in the MM consumption chain
Scattered work on emotions in RecSys
Unifying framework (too ambitious?)
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 1
Content application
Give conten
t
time
Entry stage Consumption stage Exit stage
Give recommendati
ons
choice
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 2
Content application
Entry mood
Detect entrymood
Give conten
t
Exit mood
time
Entry stage Consumption stage Exit stage
Give recommendati
ons
choice
• Context• Decision making• Influence• Diversification
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 3
Content application
Entry mood
Detect entrymood
Give conten
t
Content-induced affective state
Observe user
time
Entry stage Consumption stage Exit stage
Give recommendati
ons
choice
• Affective tagging• Affective user profiles
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 3
Content application
Entry mood
Detect entrymood
Give conten
t
Content-induced affective state Exit mood
Observe user
time
Entry stage Consumption stage Exit stage
Give recommendati
ons
choice
Detect exit
mood
• Implicit feedback• Evaluation metrics
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 3
Content application
Entry mood
Detect entrymood
Give conten
t
Content-induced affective state Exit mood
Observe user
time
Entry stage Consumption stage Exit stage
Give recommendati
ons
choice
Detect exit
mood
• Implicit feedback• Evaluation metrics
• Affective tagging• Affective user profiles
• Context• Decision making• Influence• Diversification
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART V: Affect in the decision making step
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART V: Affect in the decision making step Stage 2 and 3 are straightforward Stage 1 is interesting = new research avenues
Content application
Entry mood
Detect entrymood
Give content
Content-induced affective state Exit mood
Observe user
time
Entry stage Consumption stage Exit stage
Give recommendations
choice
Detect exit
mood
• Implicit feedback• Evaluation metrics
• Affective tagging• Affective user profiles
• Context• Decision making• Influence• Diversification
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
From data-centric to user-centric The community is problem-solving oriented
– „The existing datasets are real, why building synthetic ones?“ (??, RecSys 2011)
The data-centric approach is still rooted in the research community:– „It‘s about music, not about recommenders“ (?? at
RecSys 2011) Solving existing problems is only a part of research ...
... the other part is generating new knowledge (on how the world works) ...
... which in turn generates new problems ...
... which in turn opens new publishing/funding/citing possibilities
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
General user modeling framework Data-centric = uses data that
– Is available (genres, actors, directors ...)– Easy to acquire (rating, „liking“ ...)
But NOT necessarily data that carry information
USER MODEL
Controlled variables
Uncontrolled variables
Prediction accuracy
?
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LET‘S MOVE FORWARD Try new models! Generate new kind of data! Find out how the world really works!
Model DECISION MAKING:– Ajzen model (Andrej‘s talk)– Kahneman model
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
System 1 / System 2 (c) Kahneman, 2003
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Decision Making Modeling in RecSys
System 1 model System 2 model
Aggregation
Decision prediction
Emotiondetection
Personalitydetection Affective stimuli
detection
Contentmetadata
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
SoA Modeling in RecSys
System 1 model System 2 model
Aggregation
Decision prediction
Emotiondetection
Personalitydetection Affective stimuli
detection
Contentmetadata
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Conclusions
RecSys = decision making predictor
Assumption = emotions do influence
Scattered work Unifying framework
Our wish = Focus on stage 1: decision making:– System 1 / System 2 modeling
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Future work
Improve models
Generate dataset
Validate