User Engagement - A Scientific Challenge
-
Upload
mounia-lalmas -
Category
Technology
-
view
3.019 -
download
0
Transcript of User Engagement - A Scientific Challenge
User Engagement – A Scientific Challenge
Mounia Lalmas
Yahoo! Research Barcelona
Ioannis ArapakisRicardo Baeza-YatesGeorges DupretJanette LehmannLori McCay-PeetVidhya NavalpakkamElad Yom-Tov
Collaborators
Outline
Motivation and DefinitionCharacteristics of User EngagementCurrent measurements
Vision and focus
Two “opposite” studies1. Models of user engagement
2. Saliency and user engagement
Motivation
User engagement is a quality of user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology.
Successful technologies are not just used, they are engaged with.
A web interface that is boring, a multimedia presentation that does not captivate users’ attention or an online forum that fails to engender a sense of community are quickly dismissed with a simple mouse click.
(O’Brian and Toms, 2008)
User engagement is how we nurture and build a community. (John Byrne, Business Weeks’ online editor 2009)
Would a user engage with this web site?
http://www.nhm.ac.uk/
Would a user engage with this web site?
http://www.amazingthings.org/ (art event calendar)
Would a user engage with this web site?
http://www.lowpriceskates.com/ (e-commerce – skating)
Would a user engage with this web site?
http://chiptune.com/ (music repository)
Would a user engage with this web site?
http://www.theosbrinkagency.com/ (photographer)
What is user engagement?“The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource”
“Newish” field of research stemming from the increasing interaction between users and web services and technologies
Current work focuses on two aspects What are the characteristics of user engagement? How can we measure user engagement?
-Subjective measures-Objective measures
Characteristics of user engagement (I)
Characteristics of user engagement (II)
Forrester Research – The four I’s
Measuring Engagement, Forrester Research, June 2008
Peterson etal Engagement measure – 8 indicesClick Depth Index: page viewsDuration Index: time spentRecency Index: rate at which users return over timeLoyalty Index: level of long-term interaction the user has with the site or product (frequency)Brand Index: apparent awareness of the user of the brand, site, or product (search terms)Feedback Index: qualitative information including propensity to solicit additional information or supply direct feedbackInteraction Index: user interaction with site or product (click, upload, transaction)
Peterson etal. Measuring the immeasurable: visitor engagement, WebAnalyticsDemystified, September 2008
Subjective measuresRecord a user’s perception (generally self-reported) of the technology
Two-part processes:
Increased use of crowd-sourcing based studies
Objective measuresTo overcome the subjectivity of post-experience questionnairesTo measure engagement over large populations and over time
Objective measures – Online activities Proxy of user engagement
bounce rates, CTR, days since last visit, exit rate, entrance rate, frequency, number of new visitors, number of visitors, number of returning visitors, number of bookmarks (e.g. on Delicious), number of comments posted, number of content syndication (RSS), content contribution (adding a comment, adding a review, rating, uploading an image or video), number of repeated visitors, page views, time spent (dwell time), visits per visitors, number of tags (e.g. on Flickr), number of emailed and printed stories, number of Facebook likes, number of re-tweets, number of messages sent (instant, email), number of conversions (e.g. subscribing, buying), number of Facebook fan pages, number of search queries, common paths (e.g. from front page to mail tool then exit), external comments and reviews (for products), etc …
Characteristics and measures
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
Diagnostic and what we can doDiagnostic: work exists, but fragmented. In particular: What and how to measure depend on services and goals Lack of understanding of how to relate subjective and
objective measures
What I (we) have done:1.“Towards” Models of user engagement2.Stylistics and “engagement”
Saliency, interest, attention and positive affect Front page styles and downstream engagement Automatic linking and reading experience
Page stylistics + layout +links + saliency
user engagement within and across site
Measurements and methodologies + online analytics metrics (dwell time, …) + questionnaires + crowd-sourcing - biometrics (eye tracking, mouse tracking, …)
Goals + Models of user engagement - Metrics of user engagement
Engagement types + attention + emotion
+ activity + popularity + loyalty - functionality +/- intent & interest
Focus – Two “opposite” studies
activity metricsloyalty metrics
popularity metrics
hobbiesnavigation
social media
e-commerce
magazinesport
news
search
weathermail
weekly news
…
focused attentionaffect (emotion)
saliencymodels
US
ER
EN
GA
GE
ME
NT
1. Models of user engagement 2. On saliency, attention and positive effect
1. Models of user engagement
User engagement characteristics depend on the web application: e.g. reading mail or browsing a news portal
results in different types of engagement
Diversity of engagementModels of engagement through clustering
using various criteriae.g. user types, temporal aspects
Data and Metrics
Interaction data, 2M users, July 2011, 73 US sites
Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.
Diversity in user engagement
Users and Loyalty Sites have different user groups Proportion of user groups is site-
dependent
Time and Popularity Site engagement can be periodic
or contains peaks
Engagement of a site depends on users and time
mail, social media
shopping, entertainment
media(special events)
daily activity,navigation
media,entertainment
Methodology
General models User-based models Time-based modelsDimensions
8 metrics5 user groups8 metrics per user group
weekdays, weekend8 metrics per time span
#Dimensions 8 40 16
Kernel k-means with Kendall tau rank correlation kernel
Nb of clusters based on eigenvalue distribution of kernel matrixSignificant metric values with Kruskal-Wallis/Bonferonni
#Clusters (Models) 6 7 5
Analysing cluster centroids = models
Models of user engagement
• 6 general models
• Popularity, activity and loyalty are independent from each other
• Popularity and loyalty are influenced by external and internal factors e.g. frequency of publishing new
information, events, personal interests
• Activity depends on the structure of the site
Models based on engagement metrics
interest-specific
e-commerce,configuration
periodicmedia
Models of user engagement
User-based [7 models] Models based on engagement per
user group
Time-based [5 models] Models based on engagement
over weekdays and weekend
Models based on engagement metrics, user and time
navigation game, sporthobbies,interest-specific
daily news
Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model
The groups of models describe different aspects of engagement, i.e. they are independent from each other
Relationships between models
General
User Time
General
0.00 3.50 4.23
User 3.50 0.00 4.25
Time 4.23 4.25 0.00
Groups of models are independent from each other
Example: Model mu2
[high popularity and activity in all user groups, increasing loyalty]
50% to model mt2 [high popularity on weekends and high loyalty on weekdays]
50% to model mt3 [high activity and loyalty on weekends]
Variance of Information [0,5.61]
Models of user engagement – Recap & NextUser engagement is complex and standard
metrics capture only a part of itUser engagement depends on users and timeFirst step towards a taxonomy of models of user
engagement … and associated metrics
NextInteraction between modelsInteraction between sites (complex networks)User demographics, time of the day, geo-location, etc
Focus – Two “opposite” studies
activity metricsloyalty metrics
popularity metrics
hobbiesnavigation
social media
e-commerce
magazinesport
news
search
weathermail
weekly news
…
focused attentionaffect (emotion)
saliencymodels
US
ER
EN
GA
GE
ME
NT
1. Models of user engagement 2. On saliency, attention and positive effect
2. Saliency, attention and positive affect
How the visual catchiness (saliency) of “relevant” information impacts user engagement metrics such as focused attention and emotion (affect)focused attention refers to the exclusion of
other thingsaffect relates to the emotions experienced
during the interaction
Saliency model of visual attention developed by Itti and Koch L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 10-12 (2000).
Manipulating saliency
Web page screenshot Saliency maps
salie
nt c
ondi
tion
non-
salie
nt c
ondi
tion
Study design8 tasks = finding latest news or headline on celebrity or
entertainment topicAffect measured pre- and post- task using the Positive
e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid” Affect Schedule (PANAS)
Focused attention measured with 7-item focused attention subscale e.g. “I was so involved in my news tasks that I lost track of time”, “I blocked things out around me when I was
completing the news tasks” and perceived time Interest level in topics (pre-task) and questionnaire
(post-task) e.g. “I was interested in the content of the web pages”, “I wanted to find out more about the topics that I encountered on the web pages”
189 (90+99) participants from Amazon Mechanical Turk
PANAS (10 positive items and 10 negative items)
You feel this way right now, that is, at the present moment [1 = very slightly or not at all; 2 = a little; 3 = moderately;
4 = quite a bit; 5 = extremely]
[randomize items]
distressed, upset, guilty, scared, hostile, irritable, ashamed, nervous, jittery, afraid
interested, excited, strong, enthusiastic, proud, alert, inspired, determined, attentive, active
D. Watson, L.A. Clark & A. Tellegen. Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 47 (1988).
7-item focused attention subscale (part of the 31-item user engagement scale)
5-point scale (strong disagree to strong agree)
1. I lost myself in this news tasks experience
2. I was so involved in my news tasks that I lost track of time
3. I blocked things out around me when I was completing the news tasks
4. When I was performing these news tasks, I lost track of the world around me
5. The time I spent performing these news tasks just slipped away
6. I was absorbed in my news tasks
7. During the news tasks experience I let myself go
H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis, 2008.
Study protocolMechanical Turk: Introduction/Consent, HIT acceptance
Link to external survey; entry of MTurk worker ID
Pre-task PANAS
Pre-task topic interest questions
8 Non-Salient tasks 8 Salient tasks
Self-report of time spent finding headlines
Post-task PANAS
Self-report of affect
Focused attention scale
Self-report of focused attention
Interest and task ease questions
Demographics questionnaire
Comments
Return to MTurk to submit HIT
Saliency and positive affect
When headlines are visually non-salientusers are slow at finding them, report more
distraction due to web page features, and show a drop in affect
When headlines are visually catchy or salientuser find them faster, report that it is easy to focus,
and maintain positive affect
Saliency is helpful in task performance, focusing/avoiding distraction and in maintaining positive affect
Saliency and focused attentionAdapted focused attention subscale from the online
shopping domain to entertainment news domain
Users reported “easier to focus in the salient condition” BUT no significant improvement in the focused attention subscale or differences in perceived time spent on tasks
User interest in web page content is a good predictor of focused attention, which in turn is a good predictor of positive affect
Saliency and user engagement – Recap & Nextinteraction of saliency, focused attention, and
affect, together with user interest, is complexNext:
include web page content as a quality of user engagement in focused attention scale
more “realistic” user (interactive) reading experiencebio-metrics (mouse-tracking, eye-tracking, facial
expression, etc)
L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affecr and focused attention, CHI 2012
The big picture ……….… my vision
Page stylistics + layout +links + saliency
user engagement within and across site
Measurements and methodologies + online analytics metrics (dwell time, …) + questionnaires + crowd-sourcing - biometrics (eye tracking, mouse tracking, …)
Goals + Models of user engagement - Metrics of user engagement
Engagement types + attention + emotion
+ activity + popularity + loyalty - functionality +/- intent & interest