User Interfaces for Exploring Large Web-based Collections: Recommender Systems & Metadata-Based...
-
Upload
virgil-day -
Category
Documents
-
view
215 -
download
1
Transcript of User Interfaces for Exploring Large Web-based Collections: Recommender Systems & Metadata-Based...
User Interfaces for Exploring Large Web-based Collections: Recommender Systems & Metadata-Based
Search Interfaces
Rashmi SinhaSIMS, UC Berkeley
Structure of Talk• Broad comparison of Recommender Systems &
Search/Browse Systems
• Different User Needs: Levels of control, task (how open-ended), level of personalized need.
• Focus on Recommender Systems: 2 studies, some design suggestions
• Quick Overview of Flamenco: A Metadata-Based Search / Browse Interface
• Summary, Conclusions & Future Plans
Interaction Paradigm of Recommender Systems…
Basic Interaction Paradigm of Recommender Systems
Which book should I read?
Input (Ratings of Books): I recently enjoyed: Of Mice and Men, Bias, The Summons & Good to Great
Output (Recommendations): Books you might enjoy are…
Popularity of Recommender Systems…
I know what you’ll read next summer (Amazon, Barnes&Noble)
• what movies you should watch… (Reel, RatingZone, Amazon)
• what music you should listen to… (CDNow, Mubu, Gigabeat)
• what websites you should visit (Alexa)
• what jokes you will like (Jester)
• & who you should date (Yenta)
At the heart of Recommender Systems are Collaborative Filtering Algorithms that rely on correlation between individuals
• Meg & James: correlation = .52
• Meg & Jim: correlation = -.67
• Meg & Nick: correlation = .23
Ratings of Books 1 2 3 4 5 6 7 8
Meg 5 3 3 4 2 1
Jim 3 4 2 3 4 5 1 3
Nick 4 3 1 2 4 2 4 1
James 4 2 1 3 4 1 5 5
Recommendations For Meg
Example Recommender System…
Amazon’s Interaction Paradigm
Contrast interaction with Search /Browse Systems
Types of user needs in Exploring Collections
Looking for some new music…I am tired of my old records, looking for something new. I
know what I like, don’t know what else I might like, am open to ideas
-more open-ended-often about “individual taste”
Example Search/Browse System…
Looking for some recipes for cooking
I have some needs, (vegan dessert for 6 people). Also I have some strawberries lying around.-less open-ended
-less about “individual taste”
Integrated Browse-Search Interaction Paradigm
Chose tomatoes…
Currently refined by Course/Meal. You can change toPreparation, Cuisine, Season. System informs you what the next choices are, and what subset of items will be left if you take that path
User Experience for such systems…
User Experience in such Search/Browse interfaces
•More of a controlled experience•Every movement (forward, making a turn, backwards) is a conscious choice. (need information at every step)•User might make mistakes, and retract (go back) a step or two or start again. Each of these is a conscious choice.Experience is similar to
driving a car…
User Experience with Recommender Systems
•-user has less control over specifics of the interaction.•System does not provide information about specifics of action•-more of the black box model (some input from user, output from systems).
Experience is more similar to riding a roller coaster…
What does this mean for Recommender System Interfaces?
Research on Recommender Systems has mostly focused CF Algorithms
Collaborative Filtering Algorithms
Output (Recommendations)
Input from user
Social Recommendations
Taking a closer look at the Recommendation Process
Input User incurs cost in using system:
Time, Effort, Privacy Issues
Receives Recommendation
Cost in reviewing recommendations
Benefit if recommended item appeals
Judges if he/she will sample recommendation
Amazon’s Recommendation Process
Input: One artist/author name
Output: List of Recommendations Explore / Refine Recommendations
Search usingRecommendations
Book Recommendation Site: Sleeper
Input: Ratings of 10 books for all users Use of continuous Rating Bar
(System designed by Ken Goldberg)
Output: List of items with brief information about each item
Degree of confidence in prediction
Sleeper: Output
Study 1: Book and Movie Recommender Systems
Three book systems Amazon Books Sleeper Rating Zone
Three Movie Systems MovieCritic Amazon Movies Reel
Study design similar as before. Recommendations were sampled during study this time.
Study 2: Looking for music online:Music Recommender Systems
Five systems CDNow Amazon SongExplorer MoodLogic MediaUnbound
Not an experiment, but designed like one. Conducted in Lab environment
Broad overview to start with, then zero in on some systems
Meshing of quantitative and qualitative methods (one informing the other)
Pre-test, pre-test, pre-test
User motivation ascertained before study
Within-subjects design used wherever possible
Multiple small studies, rather than one big study
General Testing Methodology
Comprehensive Data Collection: Observation, Behavior logging with time stamps, questionnaires, post-test interviews.
Testing Methodology cont.
The Slim Logger: Simple Excel Based tool for recording timed observations.
For each of online systems:Rated items Reviewed and evaluated recommendation setCompleted questionnaire
For Study 1: also reviewed and evaluated sets of recommendations from 3 friends each
About 15-20 participants in each study, age:18 to 34 years
Study Procedure
How do recommendations from Online Systems compare to that from friends?
Popularity of Online Systems indicates that people find such systems useful. What are they useful for?
Comparing Human Recommenders (user’s friends) to Online Systems
Human Recommenders & Systems: “Good” & “Useful” Recommendations
0
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Amazon(15)
Sleeper(10)
RatingZone (8)
Friends(9)
Amazon(15)
Reel(5-10)
MovieCritic (20)
Friends(9)
Books Movies
% Good Recommendations
% Useful Recommendations
RS AverageAve. Std. Error (x) No. of Recommendations
However Users Like Online RS…
This result was supported by post test interviews.
Do you prefer recommendations from friends or online systems?
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Books Movies
Num
ber
pref
erri
ng s
yste
m
System
Friends
Why Systems Over Friends?
“Suggested a number of things I hadn’t heard of, interesting matches.”
“It was like going to Cody’s—looking at that table up front for new and interesting books.”
“Systems can pull from a large database—no one person knows about all the movies I might like.”
Recommender Systems broaden horizons
Friends mostly recommend familiar items
%Not Heard of Previously
0102030405060708090
100
Amazon Sleeper RatingZone Friends Amazon MovieCritic Reel Friends
MoviesBooks
What aspects of systems do users like?
Why do users like particular systems: Searching for reasons
Previously Liked Items & adequate Item Description are correlated.Time to Receive Recommendations & No. of Items to Rate are not correlated.
Correlation of Subjective Usefulness and Ratings of System Features
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
No. GoodRecs
Detail inItem
Description
Knowreason for
recs?
PreviouslyLiked Recs.
Time to getrecs
No of itemsto rate
co
rre
lati
on
**** **
Good, Useful & Previously Experienced Recommendations
Post Test Interviews indicate that users “trust” systems if they have already sampled some recommendations
•Previous Positive Experiences lead to “trust”
•Previous Negative Experiences lead to mistrust of system
UsefulNot yet read/
viewed
Previously read/viewed(lead to trust)
All Good Recommendations
Adequate Item Description: The RatingZone Story
0 % of Version 1 and 60% of Version 2 users found item description adequate
% Useful For Both Versions of RatingZOne
0
5
10
15
20
25
30
35
40
45
Version 1: Without Description Version 2: With Description
% U
sefu
l R
ecs
.
An adequate item description, and links to other sources about item was a crucial factor in users being convinced by a recommendation.
Study 1
System TransparencyUser perception that they understand why an item was
recommended
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Amazon Cdnow MediaUnbound
Mood Logic Song Explorer
Mea
n L
ikin
g
Not Transparent
Transparent
Transparent recommendations liked more than not-transparent ones for all five systems
Study 2
Mean liking for Familiar and Unfamiliar Recommendations
0
1
2
3
4
5
Amazon Cdnow MediaUnbound
Mood Logic Song Explorer
Mea
n L
ikin
g
Unfamiliar
Familiar
Familiar recommendations liked more than unfamiliar ones for all five systems
Some Results: Effect of familiarity on liking
Study 2
Two Models of Recommender System Success
• Recommendations from Amazon received highest liking rating for Study 1 (for books & movies) and second highest for Study 2 (Music)
• Recommendations from MediaUnbound outperformed Amazon in Study 2 (Music)
• Both systems were well liked but differed dramatically in interaction style…
Genre Selection
Favorite artist / band
Rating some songs
Amazon’s bare-bones recommendation process
Genre Selection
Media-Unbound’s long, extended (35 questions) recommendation process
Level of Familiarity
Feedback atEvery stage
Rating some songs
More feedback about user’s tastes
Setting system expectations
Users find MediaUnbound recommendations more useful
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
Useful Use in Future
% U
sers
Amazon
MediaUnbound
Also, most users preferred MediaUnbound over Amazon
But whose recommendations would they buy?
0%
5%
10%
15%
20%
25%
30%
35%
No Action Borrow /Get forfree
Buy
pe
rce
nta
ge
Amazon
Media Unbound
But, users express more interest in buying Amazon recommendations
Amazon: •Safe, conservative approach to recommendations•Recommendations are familiar, few new items•Users find system logic transparent•Users don’t feel like they learnt anything news
MediaUnbound: •Verifies from user how familiar they want recommendations•Long input process seems to generate trust•Recommendations are often new, but well liked
Discussion & Design Suggestions
Justify your Recommendations
– Adequate Item Information: Providing enough detail about item for user to make choice
– System Transparency: Generate (at least some) recommendations which are clearly linked to the rated items
– Explanation: Provide an Explanation, why the item was recommended.
– Community Ratings: Provide link to ratings / reviews by other users. If possible, present numerical summary of ratings.
Accuracy vs. Less InputDon’t sacrifice accuracy for the sake of generating quick recommendations. Users don’t mind rating more items to receive quality recommendations.
–Multilevel recommendations: Users can initially use the system by providing one rating, and are offered subsequent opportunities to refine recommendation
–Provide a happy medium between too little input (leading to low accuracy) and too much input (leading to user impatience)
–Unlike with Search Engines, users are not willing to try again and again.
Users like Rec. Systems as they provide information about new, unexpected items.
List of recommended items should include new items which the user might not find out in any other way.
List could also include some unexpected items (e.g., from other topics / genres) which the user might not have thought of themselves.
Include some New Unexpected Items
Users (especially first time users) need to develop trust in the system.
Trust in system is enhanced by the presence of items that the user has already enjoyed.
Generating some very popular (which have probably been experienced previously) in the initial recommendation set might be one way to achieve this.
Trust Generating Items
Trust Generating Items: A few very popular ones, which the system has high confidence in
Unexpected Items: Some unexpected items, whose purpose is to allow users to broaden horizons.
Transparent Items: At least some items for which the user can see the clear link between the items he /she rated and the recommendation.
New Items: Some items which are new /just released
The right mix of items
Question: Should these be presented as a sorted list / unsorted list/ different categories of recommendations?
Verify degree of familiarity user wants
This can help produce the right mix of items for each user.
To sell as many items as possible or to help users explore their tastes?
The two goals are often contradictory, at least in short term.
Important for system designer to keep goals in mind while designing system.
What kind of a system do you want?
Onwards to Search/Browse Systems…
Flamenco: Designing an innovative Search / Browse
System for Architectural Images • System supports explorations of a large
architectural image dataset
• Our goal was to build a flexible navigation and search using faceted metadata
The Philosophy• Information architecture should be designed to
integrate search throughout• Search results should reflect the information
architecture, supporting an interplay between navigation and search
• This supports the most common human search strategies.
• Use metadata to show user where to go next. More flexible than canned links
• Allow users to expand as well as refine
An Important Search Strategy
• Do a simple, general search / browse– Gets results in the generally correct area
• Look around in the local space of those results• If that space looks wrong, start over
– Akin to Shneiderman’s overview + details
• Our approach endeavors to supports this search strategy
Questions we are trying to answer
• How can Faceted Metadata be effectively used to build an easy to use Search / Browse system?– How many facets should be displayed at once?– Should facets be mixed and matched?– How much is too much?
• How should hierarchies be revealed (progressively, one step at a time)?
• How should large categories be displayed?
• How should refinement & expansion of query be supported?
Image Dataset
Architect
Image
Geo-Region Time/Date
~40,000 images, 9 hierarchical facets, rich faceted hierarchical data
Facets
Development Timeline• Needs assessment.
– Interviewed architects and conducted contextual inquiries. • Lo-fi prototyping.
– Showed paper prototype to 3 professional architects.• Design / Study Round 1.
– Simple interactive version. Users liked metadata idea.• Design / Study Round 2:
– Developed 4 different detailed versions; evaluated with 11 architects; results somewhat positive but many problems identified. Matrix emerged as a good idea.
• Design / Study Round 3. – New version based on results of Round 2– Highly positive user response
Architects’ Image Use• Common activities:
– Use images for inspiration• Browsing during early stages of design
– Collage making, sketching, pinning up on walls– This is different than illustrating PowerPoint
• Maintain sketchbooks & shoeboxes of images
• No formal organization scheme– None of 10 architects interviewed about their image
collections used indexes
• Do not like to use computers to find images
Type of tasks we want to support• Your firm wants to enter a competition to design a new central
library in downtown Oakland. Find images to support your ideas for creative use of space in crowded urban setting (the site is surrounded by skyscrapers).
• You're preparing an exhibit to show off the possibilities of "environmentally-friendly" design at an upcoming home & garden show. Find some images that will encourage new-home purchasers to consider building in this way.
• Your team is doing a pool / pool-house / garden renovation. The lead architect’s preliminary design calls for rough/unfinished materials such as concrete and iron. The client is confused and resistant. Please find 2 or 3 images to use for a collage to help the client explore the idea of “concrete in the garden.”
The Interface• Nine hierarchical facets
– Matrix– SingleTree (control interface inspired by Yahoo)
• Chess metaphor– Opening– Middlegame– Endgame
• Tightly Integrated Search• Expand as well as Refine• Intermediate pages for large categories
Begin Game for SingleTree
Middle Game for SingleTree
Chose “cultural landscapes” in structure types
Begin Game for Matrix
Next: Group by Location
Middle Game for Matrix
Next: Click on one image
Next: Expand from image view
Searched for “brick”
Next: Chose pitched brick vaults
Next: Expand using breadcrumbs
Results of User Study with 19 architects
• Users rated Matrix more highly for:– Usefulness for design work– Seeing relationships between images– Flexibility– Power
• On all except “find this image” task, users also rated the Matrix higher for:– Feeling “on track” during search– Feeling confident about having found all relevant
images
User Comments - Matrix• “Powerful at limiting and expanding result sets. Easy to
shift between searches.”
• “Keep better track of where I am located as well as possible places to go from there.”
• “Left margin menu made it easy to view other possible search queries, helped in trouble-shooting research problems.”
• “Interface was friendlier, easier, more helpful.”
• “I understood the hierarchical relationships better.”
User Comments – Single Tree
• Pro– “Simple”– “More typical of other search engines I’d use”– “Visually simpler and more intuitive…Matrix a bit
overwhelming with choices.”
• Con– “I found SingleTree difficult to use when I had to
refine my search on a search topic which I was not familiar with. I found myself guessing.”
– “SingleTree required more thought to use and to find specific images.”
– “I do not trust my typing and spelling skills. I like having categories.”
Feature Usage (%) Types of Actions
Action Categories
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00%
Refine search(reduce # of
results)
Expand search(increase # of
results)
Arrange results
Start over/backup
Matrix
Tree
Feature Usage (%) Refining
Use of Features to Refine Search
0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00%
Drill above images
Drill in matrix
Drill from image detail
Drill from large category
Drill by clicking "All N items"
Search within
Disambiguate keyword search
Matrix
Tree
Feature Usage – Expanding / Starting Over
Use of Features to Expand Search / Start Over
0.00% 5.00% 10.00% 15.00% 20.00% 25.00%
Expand searchusing breadcrumbs
Expand by clickingX
Expand from imagedetail
Go back to startmid-search
Search all, mid-task
Back
Matrix
Tree
Summary & Conclusions• A new approach to web site search
– Use hierarchical faceted metadata dynamically, integrated with search• Many difficult design decisions
– Iterating and testing was key
• Design Challenges– How to seamlessly integrate metadata previews with search (Show search results in metadata
context)– How to show hierarchical metadata from several facets (The “matrix” view, Show one level of depth
in the “matrix” view)– How to handle large metadata categories (Use intermediate pages)– How to support expanding as well as refining (Still working on it to some extent)
Overall Summary
• Two modes of information exploration…
User has more control…-Want information at every step-Flexible way to refine, change, expand
User has less control…-Wants transparency-Wants new information-Needs to be convinced that recommendation is good
Conclusions & Future Plans• Recommender Systems and Search/Browse
systems support two different modes of information exploration– Directly compare both in controlled study– Expert / Novice differences in suitability of the two
interfaces– Task based differences in suitability of interfaces
• Recommender Systems: – Role of explicit community as compared to automated
recommendations– How to integrate community with recommendations
• What about Product Advisors? Where do they fit in?
Recommender Systems ProjectKirsten SwearingenSIMS, UC Berkeley
For more informationsims.berkeley.edu/~sinha
Metadata Based Search Users InterfacesMarti Hearst
Ame Elliott, Jennifer English, Kirsten Swearingen, Ping Yee
COLLABORATORS