The Effect of Different Set-based Visualizations on User Exploration of Recommendations

The Effect of Different Set-based Visualizations on User Exploration of

Recommendations

Katrien Verbert, KU Leuven

Denis Parra, PUC Chile

Peter Brusilovsky, University of Pittsburgh

IntRS Workshop at RecSys 2014, Foster City, CA, USA

Outline

• Context of this Work in RecSys research

• Set-based Visual Interfaces for User Exploration

– TalkExplorer: Multimode graph

– SetFusion: Venn diagram

• Meta-Analysis

• Summary & Future Work

10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 2

INTRODUCTIONRecommender Systems: Introduction & Motivation

3

* Danboard (Danbo): Amazon’s cardboard robot, in these slides represents a recommender system

*

Recommender Systems (RecSys)

Systems that help people (or groups) to find relevant items in a crowded item or information space (McNee et al.

2006)


Challenges of RecSys Addressed HereTraditionally, RecSys has focused on producing accurate recommendation algorithms. In this research, we address these challenges:

1. HCI: Implementation of visualizations that enhance users’ exploration of the items suggested.

2. Recommendation Tasks: Tackling exploration of recommendations,not only rating prediction or Top-N.

3. Meta-Analysis: Comparing results of different studies to generalizeresults.


Research Platform

• The studies were conducted using Conference Navigator, a Conference Support System

• Our goal was recommending conference talks


Program Proceedings Author List Recommendations

http://halley.exp.sis.pitt.edu/cn3/

RELATED WORK OF VISUAL INTERFACES FOR RECSYS

Previous research related to this work / Motivating results from TalkExplorer study

10/06/2014Verbet, Parra, & Brusilovsky .~ IntRs as

RecSys 20147

PeerChooser – CF movies


O'Donovan, J., Smyth, B., Gretarsson, B., Bostandjiev, S., & Höllerer, T. (2008, April). PeerChooser: visual interactive recommendation

SmallWorlds – CF Social


Gretarsson, B., O'Donovan, J., Bostandjiev, S., Hall, C., & Höllerer, T. (2010, June). Smallworlds: Visualizing social recommendations.

TasteWeights – Hybrid Recommender


Bostandjiev, S., O'Donovan, J., & Höllerer, T. (2012, September). Tasteweights: a visual interactive hybrid recommender system

TALKEXPLORER: A GRAPH-BASED INTERACTIVE RECOMMENDER

11

TalkExplorer – IUI 2013

• Adaptation of Aduna Visualization to CN

• Main research question: Does fusion (intersection) of contexts of relevance improve user experience?


TalkExplorer - I


EntitiesTags, Recommender Agents, Users

TalkExplorer - II


RecommenderRecommender

Cluster with

intersection

of entities

Cluster (of talks)

associated to only

one entity

• Canvas Area: Intersections of Different Entities

User

TalkExplorer - III


ItemsTalks explored by the user

Our Assumptions

• Items which are relevant in more that one aspect could be more valuable to the users

• Displaying multiple aspects of relevance visually is important for the users in the process of item’s exploration


TalkExplorer Studies I & II

• Study I– Controlled Experiment: Users were asked to discover

relevant talks by exploring the three types of entities: tags, recommender agents and users.

– Conducted at Hypertext and UMAP 2012 (21 users)

– Subjects familiar with Visualizations and Recsys

• Study II– Field Study: Users were left free to explore the interface.

– Conducted at LAK 2012 and ECTEL 2013 (18 users)

– Subjects familiar with visualizations, but not much with RecSys


Evaluation: Intersections & Effectiveness

• What do we call an “Intersection”?

• We used #explorations on intersections and their effectiveness, defined as:

Effectiveness = |𝑏𝑜𝑜𝑘𝑚𝑎𝑟𝑘𝑒𝑑 𝑖𝑡𝑒𝑚𝑠|

|𝑖𝑛𝑡𝑒𝑟𝑒𝑠𝑒𝑐𝑡𝑖𝑜𝑛𝑠 𝑒𝑥𝑝𝑙𝑜𝑟𝑒𝑑|


Results of Studies I & II

• Effectiveness increases with intersections of more entities

• Effectiveness wasn’t affected in the field study (study 2)

• … but exploration distribution was affected


Drawback: Visualizing Intersections

Clustermap Venn diagram

• Venn diagram: more natural way to visualize intersections


SETFUSION: VENN DIAGRAM FOR USER-CONTROLLABLE INTERFACE

21

SetFusion – IUI 2014


SetFusion I


Traditional

Ranked List

Papers sorted by Relevance. It combines 3 recommendation approaches.

SetFusion - II


SlidersAllow the user to control the importance of each data source or recommendation method

Interactive Venn DiagramAllows the user to inspect and to filter papers recommended. Actions available:- Filter item list by clicking on an area- Highlight a paper by mouse-over on a circle- Scroll to paper by clicking on a circle- Indicate bookmarked papers

SetFusion – UMAP 2012

• Field Study: let users freely explore the interface


- ~50% (50 users) tried the SetFusion recommender

- 28% (14 users) bookmarked at least one paper

- Users explored in average 14.9 talks and bookmarked 7.36 talks in average.

A AB ABC AC B BC C

15 7 9 26 18 4 17

16% 7% 9% 27% 19% 4% 18%

Distribution of bookmarks per method or combination of methods

META-ANALYSISDescription and Analysis of the results of the 3 user studies

TalkExplorer vs. SetFusion

• Comparing distributions of explorations


In studies 1 and 2 over talkEplorer we observed an important change in the distribution of explorations.

TalkExplorer vs. SetFusion

• Comparing distributions of explorations


Comparing the field studies:- In TalkExplorer, 84% of

the explorations over intersections were performed over clusters of 1 item

- In SetFusion, was only 52%, compared to 48% (18% + 30%) of multiple intersections, diff. not statistically significant

CONCLUSIONS & FUTURE WORK

Summary of this Talk

• We presented two implementations of visual interactive interfaces that tackle exploration on a recommendation setting

• We showed that intersections of several contexts of relevance help to discover relevant items

• The visual paradigm used can have a strong effect on user behavior: we need to keep working on visual representation that promote exploration without increasing the cognitive load over the users


Limitations & Future Work

• Apply our approach to other domains (fusion of data sources or recommendation algorithms)

• For SetFusion, find alternatives to scale the approach to more than 3 sets, potential alternatives:

– Clustering and

– Radial sets

• Consider other factors that interact with the user satisfaction:

– Controllability by itself vs. minimum level of accuracy


THANKS!QUESTIONS? [email protected]

Mixed Hybridization: Item Score


M: is the set of all methods available to fuserankreci,mj : rank–position in the list of a recommended item reci : recommended method imj, : recommendation method jWmj : weight given by the user to the method mj using the controllable interface |Mreci| represents the number of methods by which item reci was recommended

Slider weight

Hybridization Methods (Burke 2002)


Hybridization Description

Weighted The scores (or votes) of several recommendation techniques are

combined together to produce a single recommendation.

Switching The system switches between recommendation techniques depending

on the current situation.

Mixed Recommendations from several different recommenders are

presented at the same time

Feature

combination

Features from different recommendation data sources are thrown

together into a single recommendation algorithm

Cascade One recommender refines the recommendations given by another.

Feature

augmentation

Output from one technique is used as an input feature to another.

Meta-level The model learned by one recommender is used as input to another.

The Effect of Different Set-based Visualizations on User Exploration of Recommendations

Technology

Transcript of The Effect of Different Set-based Visualizations on User Exploration of Recommendations