Post on 19-Dec-2015
making sense on
small screens
patrick baudisch
microsoft research
adaptive systems and interaction
interaction focus
small screensstore content off-screen
halo
store content in another layer
multiblending
squeezing it in
fishnetmake the unreadable readable
summary thumbnailsmanually collapse irrelevant material
collapse-to-zoom
(making two small screens into one big
stitching, mouse ether) collapse
expand
return
scenarios…
• commuting between Seattle & Redmond• mobile user interface supports
• recall: bookmarked weather report site• execute: navigation system
• the real work I do on my desktop• PC cell phone sync cable
but not for all…
• PCs: 300 million
• cell phones: 1.5 billion
…the “others”
• rural computing, such as India• phone users, never had access to PC• phone = first access to the digital world
problems
• 1. phone-only usersneed to• plan and understand
maps and routes• explore the web interactively• analyze and compare data
(sensemaking)• but almost all apps designed
for phone+PC combo users
• 2. screen too small• most content designed for desktop PC
[rooms, card et al 86]
research agenda
• long term goalhow much and what functionality can/should we offer phone only users?
• short term goalovercome limitations of small screen size
oh, and, does it matter?
• PCs market• at 300 million PCs fairly saturated
• cell phones• already 5 times bigger• growing at high speed
related work
skip
related work: zooming
• [Xie etc. al, www’04]:tap to zoom into a tile
related work: overview+detail
• [O’Hara et. at CHI 99]: readable text on hover
related work: multiple foci• [Wobbrock
et. al UIST’02]: Web thumb
related work: popouts
• popout prism [suh, et al., chi’02]
• enhanced thumbnails
related work: fisheyes
• document lens [robertson, uist’93]• unifying presentation space [carpendale, uist’01]• focus+context sketching on a Pocket PC [lank, chi’04]
fishnet,collapse,s-thumb
related work: peep hole
• Ka-Ping Yee [CHI’03]
collapse
expand
add off screen
squish context
make readable
collapse irrelevant
add a layer
halo
halo
store content in periphery
the problem
+
halo <demo>
cinematography
1. entry and exit points
2. point of viewarrow-based techniques
3. partially out of the frame halo
rings are familiar, graceful degradation
streetlamps
• aura visible from distance• aura is round• overlapping auras aggregate• fading of aura indicates distance
what we changed• smooth transition sharp edge• disks rings• dark background light background
title: user study
user study
interfacesarc/arrow fading offscale 110-300m/cmmap as backdropreadability oksame selectable
size
hypothesis:
halo faster
halo ring distance from display border
legend
tasks
(a) locate (b) closest
(d) avoid(c) traverse
procedure
• 12 participants• within subject design, counterbalanced• four training maps per interface/task,
then eight timed maps• questionnaire
task completion time
Task Arrow interface Halo interface
Locate 20.1 (7.3) 16.8 (6.7)
Closest 9.9 (10.1) 6.6 (5.3)
Traverse 20.6 (14.1) 16.8 (8.7)
Avoid 9.2 (4.7) 7.7 (5.8)
0
5
10
15
20
25
Locate Closest Traverse Avoid
Arrow interface
Halo interface
33%
16%
error rateTask Arrow interface Halo interface
Locate 23.5 pixels (21.6) 28.4 pixels (33.8)
Closest 22% (42%) 21% (41%)
Traverse 97.4 pixels (94.7) 81.0 pixels (96.7)
Avoid 15% (35%) 14% (34%)
0
5
10
15
20
25
30
Locate Closest Traverse Avoid
Arrow interface
Halo interface
participants underestimated distances by 26% participants saw ovals (gestalt laws?) we can compensate for that: width += 35%
0
1
2
3
4
5
6
7
8
9
Locate Closest Traverse Avoid
Arrow interface
Halo interface
subjective preference
results
• halo 16%-33% faster than arrows• no split attention• distortion-free space• scale independent• no need to annotate distance• perceive all rings at once
[treisman & gormican]
• limitation: max number or rings
> back to large displays
teaser: multiblending
• application UI competes with content for screen space
• where to put it?
> back to large displays
multiblending
collapse
expand
add off screen
squish context
make readable
collapse irrelevant
add a layer
conclude
multiblending: summary
solution multiblending:glass palette distinguished from background photo& background more recognizable
solution multiblending:glass palette distinguished from background photo& background more recognizable
problem with traditional alpha blending: is thisbush in palette or background?
problem with traditional alpha blending: is thisbush in palette or background?
goal of (alpha)-blending
• goal: get twice the benefitfrom limited screen space
• applications of alpha palettes • everquest• but not photoshop
• example• overview: explain detail view• photo: context for editing
limitations of alpha-blending
• limitations• loss of contrast in both layers• colors are not “true” anymore• interference makes it unclear
which layer a feature belongs to
• all features are affected
• reason• each channel (rgb)
= weighted sum of respective channel in source images
multiblending
• preserve the most relevant features
allow different weight for each features class
• avoid visual ambiguity
to preserve feature in one layer eliminate that feature in other layers
use perception-oriented feature classes (CIE Lab color + edges)
glass palette: steps 1-2
1. desaturate 2. emboss…
glass palette: step 3
3. blurbackground…
problems withnoisy background
glass palette: step 4
4. removeunused areas
unuseddecoration
step 5 & limitation
5. …re-map features
collision: layers with same requirements
opaque
alphablending
complete example
multiblending
implementation
• recent graphics cards (with pixel shader 2.0) does computation on the fly
• 145 frames/sec
general procedure
• glass palettes is just one possible example
• other application scenarios may favor different palette styles
• general procedure: for each feature class• decide which layer benefits more from it• eliminate feature from other layer
or map it to a different feature class
user studies
• compare multiblending with alpha blending at several levels of opacity
background visibility
palette
visibility
better technique
background visibility
background visibility
0.23
0.33
0.42
0.25
0.34
0.45 0.46
0.15
0.0
0.1
0.2
0.3
0.4
0.5
0.6
alpha-10 alpha-25 alpha-50 glass
erro
r am
ount
(st
eps)
light featuresdark featureslight featuresdark features
palette visibility
palette visibility
0.38
0.02 0.01 0.01
1.58
0.05 0.02 0.06
0.0
0.5
1.0
1.5
2.0
2.5
alpha-10 alpha-25 alpha-50 glass
erro
rs p
er t
rial
over light backgroundover dark backgroundover light backgroundover dark background
results
• multiblending• is a generalization of alpha blending• provides both background and foreground visibility
simultaneously (no single alpha value can do this)• avoids interference• allows preserving the most relevant features
• is widely applicable• also for desktop: chameleon skin, games (Everquest)…
ok, back to large documents: web pages
ok, back to large documents: web pages
fishnet
collapse
expand
add off screen
make readable
collapse irrelevant
add a layer
conclude
squish context
fishnet
squish context
summary: highlighted search
US
Blackout
PGA
Microsoft
found!
found!
don’t know
don’t know
• highlights tell me “found”• …but cannot tell me “not there”
fishnet
not there!not there!
found!found!
benefit: judge relevanceof web page at a glance
benefit: judge relevanceof web page at a glance
Microso
design
scrolling, context highlighting
space fillingspace filling
use parallel projection
manhattan lens zoomscapes central projection parallel projection
readabilitybottlenecks
contentdisrupted
tall objectswon’t shrink
• two tasks…thus two resolutions• reading: render as much as possible at full size• skimming: preserve readability of context as well as possible
• homogenous context resolution preserves alignment
space filling
context area: popouts
• ensure that search terms are readable
• preserves font attribute, such as underline
• shows anchor
extended highlights
• implementation: place color-full table cell behind
popout prism:popouts interfere(need to turn off to read)
fishnet:similar saliency withextended highlights
anchoring
focus area
top contextcontext
bottom context
anchor points popout
user study
user study
• goal: when to use fisheye view; when not to use it 4 tasks
• 3 interfaces with comparable functionality• linear: standard internet explorer• overview: left 25% of screen = overview• fisheye: top & bottom 25% = context
• size & aspect ration:• same amount of screen space• fit in professional web pages, 800 pixels wide with overview/context: different aspect ratios
• 13 participants (7 males, 6 females)
task 1: In this page outdated? babydisney video sale
task 2:productchoice
512 MB
DVD
XP Home
task 3: co-occurrenceof and Mariners Mets
task 4:textanalysis
Clinton
… and hypotheses
distinguishing columns immaterial …
…necessary
distinguishing rows immaterial
favors overview-fisheye favorsfisheye
… necessaryfavors overview favors linear view
outdated task
product comparison
analysis
co-occurrence task
02468
101214161820
productchoice
co-occur-rence
is this pageout-dated?
analysistas
k c
om
ple
tio
n t
ime
(in
sec
)
Linear Overview Fisheye
results
confirmed hypotheses
subjectivepreference
• per interface + task• consistent with task time• exception: overview preferred
for the co-occurrence task
• however overall• 10 participants preferred overview,
only 3 preferred fisheye, none preferred linear• fisheye view less familiar, more cognitive effort?
02
46810
121416
1820
Out-dated Product choice Co-occurrence Analysis
Tim
e (i
n s
ec)
0
1
2
3
4
5
6
7
8
Out-datedProduct choice Co-occurrence Analysis
Linear Overview Fisheye
task completion time
satisfaction
discussion:when to use what?• fisheye views + search term
highlighting results in useful interface
• there is no black and white wrt fisheye views
• use fisheye view• if grouping in rows does not matter
• use overview• if grouping in columns does not matter
• consider fisheye & overview on demand• unlike overview, fisheye no page reflow
but wait: web pages aretoo wide for PDA
summary thumbnails
collapse
expand
add off screen
squish context
make readable
collapse irrelevant
add a layer
conclude
unreadablereadable
• any screen size• any font size
demo
implementationscale font up to 23pt2
crop text to repair3
render page at small scale4
mars mission
mars mission
mars m
bitmap
html
• load in web browser control
• for each mshtml element indocument object model (DOM)• count lines• increase font size• reduce text to preserve #lines
• client does scaling
text reduction
• current • cut off at the end of paragraph• OR remove common words
• future• use something smarter
(but what we have works surprisingly well)
• however• goal is to show users where to zoom in• may not be a summarizer
related work
user study design
• first set of “pilots” (20 students)• go to history, pick fifth page, tell us URL and task
• second set of “pilots” (2 internal)• pilots to generate balanced sets, remove 1
outlier page
• actual participants (12 externally recruited)• the actual study
results
Thumbnail SummaryThumbnail
Single Column
# o
f p
arti
cip
ants
0
2
4
6
8
10
0
5
10
15
20
25
Thumbnail Semantic Th. Single Column Desktop
Me
an
ta
sk
tim
es
(s
)
task time
preference
more results
0
10
20
30
TN ST SC DT
Tot
al v
ertic
al s
crol
l
0
10
20
30
TN ST SC DT# ve
rt s
crol
l dir
chan
ge
a
c
0
2
4
6
8
10
12
14
Thumbnail Summary Thumbnail
# o
f zo
om
ing
ev
en
ts
# zooming
scrolling
discussion
• summary thumbnails subsume traditional thumbnails
• simple conversion using a proxy server
• any platform
product
collapse-to-zoom
collapse
expand
add off screen
squish context
make readable
add a layer
collapse irrelevant
conclude
collapse-to-zoom
narrow down manually
collapse
expand
short demo
(there will be a second, slightly longer demo at the end of this talk)
?
limitation of thumbnails
• irrelevant page content costs valuable screen space
• when used with traditional thumbnails page content can become unreadable…
related work
related work
• approaches• device-specific authoring
• multiple-device authoring
• automatic re-authoring
• and client-side navigation
thumbnail viewscollapse-to-zoom
collapse-to-zoom
• allow users to use their knowledge about relevant areas zoom in
(arbitrary rectangular regions)
.
collapse-to-zoom
• allow users to use their knowledge about relevant areas zoom in
• …but also allow leveraging their knowledge about what is not relevant collapse
collapse-to-zoom
• always show full page width• use freed space to grow
remaining (relevant) content
collapse-to-zoom
• provide visual context at all times: placeholders
collapse-to-zoom
• allow bookmarking collapsed state
marquee menu:there are 4 ways to select
• today: no distinction between the four ways of opening rectangular selection
marquee menu: direction selects 1 of 4 commands
collapse-cell
expand-cell
collapse-column
expand&zoom
expand
collapse
an
mai
n
com
ds m
the name of the game
• photoshop: “marquee selection”
• “marking menu”:selecting commands with a pen stroke
• combine both marquee menu
transfer to smartphone
• generic• label cells with numeric
codes and let users enter cell label [Paek et al. CSCW 2004]
• simple: reduce to 1D• collapse column and
expand column
webTV
implementation
• runs on desktop / tabletPC• page-splitting based on DOM
• limitation: can’t start drag over link user study
collapse
expand
add off screen
squish context
make readable
collapse irrelevant
add a layer
next steps
integrating it all
collapse-to-zoomfor interaction
summary thumbnailsfits page to screen width
+ +fishnet
fits page to screen height
conclusions
• new visualizations techniquesare the foundation for viewing largedocuments, such as web pages
next steps
• support users’ sensemaking activities on small screen devices• how to pick the best insurance on a PDA?• how to author a web page on a phone?• how to do a tax return on a phone?
read more
collapse
expand
add off screen
squish context
make readable
collapse irrelevant
add a layer
CHI 2003
CHI 2004
UIST 2004
CHI 2005AVI 2004
patrickbaudisch.com/projects
thanks!
• thanks to all the coauthors• ruth rosenholtz• carl gutwin• bongshin lee• heidi lam• xing xie
• and VIBE
END
related work: fisheyes
• works well to reduce page length
• applying it to page width works not so well• forces users to scroll for
each line
fishnet [Baudisch, AVI 2004]
back to collapse-to-zoom or summary thumbnails
next visit of that page:page is already pre-collapsedwhich leads to fullyreadable detail viewUser uses expand-and-zoomgesture…User can follow links directlyfrom within overviewContent area expands moreand is now readable…make page content grown.Now collapses “menu” columnuser collapses“archive” column to…
collapse-to-zoom:
walkthrough
thumbnail view:unreadably small
next:search
1. locate task
click at expected location of off-screen targets
had tosimulate on PC
2. closest task
click arrow/arc or off-screen location closest to car
3. traverse task
4. avoid task
click on hospital farthest away from traffic jams
background visibility
50% gp
25%10%
palette visibility
palette visibility
50% gp25%10%
background visibility
50% gp
25%10%
why even consider fisheye?
• web pages contain little spatial information
• limit distortion to what is absolutely necessary
• enhance periphery with something useful:search term popouts
research question
• for what types of pages does highlighted search benefit from fisheye view?
• different from [hornbæck & frokjær, chi’01] • fishnet shows entire page• integrated navigation• search term highlighting add utility to context
areas
independent variables…
distinguishing columns immaterial …
…necessary
distinguishing rows immaterial
task 1 task 2
… necessary task 3 task 4