Marti Hearst SIMS 247 SIMS 247 Lecture 19 Visualizing Text and Text Collections March 31, 1998.
SIMS 213: User Interface Design & Development Marti Hearst Thurs, March 18, 2004.
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of SIMS 213: User Interface Design & Development Marti Hearst Thurs, March 18, 2004.
SIMS 213: User Interface Design & Development
Marti HearstThurs, March 18, 2004
Outline
How do people search for images?Current approaches:– Spatial similarity– Keywords
Our approach:– Hierarchical Faceted Metadata– Very careful UI design and testing
Usability StudyConclusions
How do people want to search and browse images?
Ethnographic studies of people who use images intensely find:– Find specific objects is easy
• Find images of the Empire State Building– Browsing is hard, and people want to use rich
descriptors.
Ethnographic Studies
Garber & Grunes ’92– Art directors, art buyers, stock photo researchers– Search for appropriate images is iterative– After specifying and weighting criteria, searchers view
retrieved images, then• Add restrictions• Change criteria• Redefine Search
– Concept starts out loosely defined, then becomes more refined.
Ethnographic Studies
Markkula & Sormunen ’00– Journalists and newspaper editors– Choosing photos from a digital archive
• Stressed a need for browsing• Searching for specific objects is trivial• Photos need to deal with themes, places, types of objects, views
– Had access to a powerful interface, but it had 40 entry forms and was generally hard to use; no one used it.
Query Study
Armitage & Enser ’97– Analyzed 1,749 queries submitted to 7 image and
film archives– Classified queries into a 3x4 facet matrix
• Rio Carnivals: Geo Location x Kind of Event– Conclude that users want to search images
according to combinations of topical categories.
Ethnographic Study
Ame Elliot ’02– Participants: Architects
Common activities:– Use images for inspiration
• Browsing during early stages of design– Collage making, sketching, pinning up on walls
• This is different than illustrating powerpoint
Maintain sketchbooks & shoeboxes of images– Young professionals have ~500, older ~5k
No formal organization scheme– None of 10 architects interviewed about their image collections used
indexes
Do not like to use computers to find images
Current Approaches to Image Search
Using Visual “Content”– Extract color, texture, shape
• QBIC (Flickner et al. ‘95)• Blobworld (Carson et al. ‘99)• Body Plans (Forsyth & Fleck ‘00)• Piction: images + text (Srihari et al. ’91 ’99)
– Two uses:• Show a clustered similarity space • Show those images similar to a selected one
– Usability studies:• Rodden et al.: a series of studies• Clusters don’t work; showing textual labels is promising.
Rodden et al., CHI 2001
Rodden et al., CHI 2001
Rodden et al., CHI 2001
Current Approaches to Image Search
Keyword based– WebSeek (Smith and Jain ’97)– Commercial image vendors (Corbis, Getty)– Commercial web image search systems– Museum web sites
A Disconnect
Why are image search systems built so differently from what people want?
– An image is worth a thousand words.– But the converse has merit too!
Some Challenges
Users don’t like new search interfaces.How to show lots more information without overwhelming or confusing?
Our Approach
Integrate the search seamlessly into the information architecture.Use proper HCI methodologies. Use faceted metadata
Example of Faceted Metadata:Medical Subject Headings (MeSH)
Facets 1. Anatomy [A] 2. Organisms [B] 3. Diseases [C] 4. Chemicals and Drugs [D] 5. Analytical, Diagnostic and Therapeutic Techniques and Equipment [E] 6. Psychiatry and Psychology [F] 7. Biological Sciences [G] 8. Physical Sciences [H] 9. Anthropology, Education, Sociology and Social Phenomena [I] 10. Technology and Food and Beverages [J] 11. Humanities [K] 12. Information Science [L] 13. Persons [M] 14. Health Care [N] 15. Geographic Locations [Z]
Each Facet Has Hierarchy
1. Anatomy [A] Body Regions [A01] 2. [B] Musculoskeletal System [A02] 3. [C] Digestive System [A03] 4. [D] Respiratory System [A04] 5. [E] Urogenital System [A05] 6. [F] …… 7. [G] 8. Physical Sciences [H] 9. [I] 10. [J] 11. [K] 12. [L] 13. [M]
Descending the Hierarchy
1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] 9. [I] 10. [J] 11. [K] 12. [L] 13. [M]
Descending the Hierarchy
1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] Electronics 9. [I] Astronomy 10. [J] Nature 11. [K] Time 12. [L] Weights and Measures 13. [M] ….
Our Approach
Integrate the search seamlessly into the information architecture.Use proper HCI methodologies. Use faceted metadata:– Generate pages from a database– More flexible than canned hyperlinks– Less complex than full search– Help users see where to go next and return to what
happened previously
Questions we are trying to answerHow many facets are allowable?Should facets be mixed and matched?How much is too much?Should hierarchies be progressively revealed, tabbed, some combination?How should free-text search be integrated?
The Flamenco Interface
Hierarchical facetsChess metaphor– Opening– Middle game– End game
Tightly Integrated SearchExpand as well as RefineIntermediate pages for large categories
What is Tricky About This?
It is easy to do it poorly– Yahoo directory structure
It is hard to be not overwhelming– Most users prefer simplicity unless complexity really
makes a difference
It is hard to “make it flow”– Can it feel like “browsing the shelves”?
Using HCI MethodologyIdentify Target Population– Architects, city planners
Needs assessment. – Interviewed architects and conducted contextual inquiries.
Lo-fi prototyping. – Showed paper prototype to 3 professional architects.
Design / Study Round 1. – Simple interactive version. Users liked metadata idea.
Design / Study Round 2: – Developed 4 different detailed versions; evaluated with 11
architects; results somewhat positive but many problems identified. Matrix emerged as a good idea.
Metadata revision. – Compressed and simplified the metadata hierarchies
Our Project History
Design / Study Round 3. – New version based on results of Round 2– Highly positive user response
Identified new user population/collection– Students and scholars of art history– Fine arts images
Study Round 4– Compare the metadata system to a strong,
representative baseline
New Usability Study
Participants & Collection– 32 Art History Students– ~35,000 images from SF Fine Arts Museum
Study Design– Within-subjects
• Each participant sees both interfaces• Balanced in terms of order and tasks
– Participants assess each interface after use– Afterwards they compare them directly
• Data recorded in behavior logs, server logs, paper-surveys; one or two experienced testers at each trial.
• Used 9 point Likert scales.• Session took about 1.5 hours; pay was $15/hour
The Baseline System
FloogleTake the best of the existing keyword-based image search systems
Comparison of Common Image Search Systems
System Collection # Results /page
Categories?
# Familiar
Google Web 20 No 27
AltaVista Web 15 No 8
Corbis Photos 9-36 No 8
Getty Photos, Art 12-90 Yes 6
MS Office Photos, Clip art
6-100 Yes N/A
Thinker Fine arts images
10 Yes 4
BASELINE Fine arts images
40 Yes N/A
swordsword
Evaluation Quandary
How to assess the success of browsing?– Timing is usually not a good indicator– People often spend longer when browsing is going
well.• Not the case for directed search
– Can look for comprehensiveness and correctness (precision and recall) …
– … But subjective measures seem to be most important here.
Hypotheses
We attempted to design tasks to test the following hypotheses:– Participants will experience greater search satisfaction, feel
greater confidence in the results, produce higher recall, and encounter fewer dead ends using FC over Baseline
– FC will perceived to be more useful and flexible than Baseline– Participants will feel more familiar with the contents of the
collection after using FC– Participants will use FC to create multi-faceted queries
Four Types of Tasks
– Unstructured (3): Search for images of interest – Structured Task (11-14): Gather materials for an art history
essay on a given topic, e.g.• Find all woodcuts created in the US• Choose the decade with the most• Select one of the artists in this periods and show all of their woodcuts• Choose a subject depicted in these works and find another artist who
treated the same subject in a different way.
– Structured Task (10): compare related images• Find images by artists from 2 different countries that depict conflict
between groups.
– Unstructured (5): search for images of interest
Other Points
Participants were NOT walked through the interfaces.The wording of Task 2 reflected the metadata; not the case for Task 3Within tasks, queries were not different in difficulty (t’s<1.7, p >0.05 according to post-task questions)
Flamenco is and order of magnitude slower than Floogle on average.– In task 2 users were allowed 3 more minutes in FC than in
Baseline.– Time spent in tasks 2 and 3 were significantly longer in FC
(about 2 min more).
Results
Participants felt significantly more confident they had found all relevant images using FC (Task 2: t(62)=2.18, p<.05; Task 3: t(62)=2.03, p<.05)
Participants felt significantly more satisfied with the results (Task 2: t(62)=3.78, p<.001; Task 3: t(62)=2.03, p<.05)
Recall scores:– Task2a: In Baseline 57% of participants found all relevant
results, in FC 81% found all.– Task 2b: In Baseline 21% found all relevant, in FC 77% found
all.
Post-Interface Assessments
All significant at p<.05 except simple and overwhelming
Perceived Uses of Interfaces
What is interface useful for?
6.44
5.475.91
4.91
7.97 7.91
6.646.16
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
Useful for mycoursework
Useful forexploring anunfamiliarcollection
Useful for findinga particular image
Useful for seeingrelationships b/w
images
SHASTA
DENALI
Baseline
FC
Post-Test Comparison
15 16
2 30
1 29
4 28
8 23
6 24
28 3
1 31
2 29
FCBaseline
Overall Assessment
More useful for your tasksEasiest to useMost flexible
More likely to result in dead endsHelped you learn more
Overall preference
Find images of rosesFind all works from a given period
Find pictures by 2 artists in same media
Which Interface Preferable For:
Facet Usage
Facets driven largely by task content– Multiple facets 45% of time in structured tasks
For unstructured tasks, – Artists (17%)– Date (15%)– Location (15%)– Others ranged from 5-12%– Multiple facets 19% of time
From end game, expansion from– Artists (39%)– Media (29%)– Shapes (19%)
Qualitative ObservationsBaseline:– Simplicity, similarity to Google a plus– Also noted the usefulness of the category links
FC:– Starting page “well-organized”, gave “ideas for what to search for”– Query previews were commented on explicitly by 9 participants– Commented on matrix prompting where to go next
• 3 were confused about what the matrix shows– Generally liked the grouping and organizing– End game links seemed useful; 9 explicitly remarked positively on
the guidance provided there.– Often get requests to use the system in future
Study Results Summary
Overwhelmingly positive results for the faceted metadata interface.Somewhat heavy use of multiple facets.Strong preference over the current state of the art.This result not seen in similarity-based image search interfaces.Hypotheses are supported.
Summary
Usability studies done on 3 collections:– Recipes: 13,000 items– Architecture Images: 40,000 items– Fine Arts Images: 35,000 items
Conclusions:– Users like and are successful with the dynamic faceted
hierarchical metadata, especially for browsing tasks– Very positive results, in contrast with studies on earlier iterations– Note: it seems you have to care about the contents of the
collection to like the interface