Jason Hong and James Landay University of California Berkeley Group for User Interface Research
description
Transcript of Jason Hong and James Landay University of California Berkeley Group for User Interface Research
![Page 1: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/1.jpg)
Nov 06 2000 1
Jason Hong and James LandayUniversity of California Berkeley
Group for User Interface Research
![Page 2: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/2.jpg)
Nov 06 2000 2
Motivation• Sketching is useful in many settings
![Page 3: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/3.jpg)
Nov 06 2000 3
• Creative / Design• SILK• Music Notepad• Knight• Elec. Cocktail
Napkin• Teddy• PatchWork
Survey of Informal Ink Apps• Capture
• ZenPad• NotePals• Dynomite• FiloChat
• Others• Pegasus• XLibris• PerSketch• Kramer's
Patches
• Whiteboard• Tivoli• Flatland
![Page 4: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/4.jpg)
Nov 06 2000 4
Example - Elec. Cocktail Napkin
Gross and Do (UIST96)
![Page 5: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/5.jpg)
Nov 06 2000 5
Example - Flatland
Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)
![Page 6: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/6.jpg)
Nov 06 2000 6
Common Features• Pen input as ink or gesture
• Stroke is pen input from dragging• Ink is a stroke that stays on-screen• Gesture is a stroke that activates a command
• Recognize ink as objects• Transformation / clean-up of ink
• Immediate / deferred processing of ink
erase me
![Page 7: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/7.jpg)
Nov 06 2000 7
Common Features • Contain objects in addition to ink• Selecting and moving objects• Grouping of objects• Layering of objects
![Page 8: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/8.jpg)
Nov 06 2000 8
Related Work• Commercial Software Support
– Newton OS– PenPoint– Microsoft Windows for Pen Computing– Windows CE– Palm OS
• Problems– Form-based and handwriting interfaces– Not easily extensible for building
informal ink apps
![Page 9: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/9.jpg)
Nov 06 2000 9
Related Work• Research Software Support
– ArtKit (Henry et. al. 1990)– Garnet Pen Int. (Landay et. al. 1993)– Patches (Kramer, 1994)– OOPS (Mankoff et. al. 2000)
• Problems– Need more reusable libraries for
handling and processing strokes– Need more extensibility so lots of kinds
of informal ink apps can be built
![Page 10: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/10.jpg)
Nov 06 2000 10
Problems• Software infrastructure support for
informal ink apps rudimentary• Pen-based interaction techniques
difficult to implement• Similar to GUI development in 1980s
– Lots of applications + interaction techniques
– Need cohesive frameworks + toolkits
![Page 11: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/11.jpg)
Nov 06 2000 11
Goals of SATIN• Toolkit support for informal ink apps
– Provide reusable mechanisms for handling + processing strokes
– Separate mechanism from policy– Be extensible for new kinds of apps
• Reusable widgets for pens• Distribute this toolkit for use
![Page 12: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/12.jpg)
Nov 06 2000 12
Overview Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions
![Page 13: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/13.jpg)
Nov 06 2000 13
DENIM
![Page 14: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/14.jpg)
Nov 06 2000 14
SketchySPICE
![Page 15: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/15.jpg)
Nov 06 2000 15
SketchySPICE
![Page 16: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/16.jpg)
Nov 06 2000 16
SketchySPICE
![Page 17: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/17.jpg)
Nov 06 2000 17
Some Metrics
SATIN DENIM Sketchy SPICE
#source files 180 76 7#methods 2192 642 63#classes 220 131 32#lines of code
20000 9000 1000
time 1 person18 months
3 people 3 months
1 person 3 days
![Page 18: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/18.jpg)
Nov 06 2000 18
Overview Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions
![Page 19: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/19.jpg)
Nov 06 2000 19
Architecture Overview
Java Core Classes
Java Virtual Machine
Java2D
SATIN
Swing
ApplicationSATIN relies on Java2D and Swing
Applications use SATIN, can use Java2D and Swing
![Page 20: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/20.jpg)
Nov 06 2000 20
Architecture OverviewSATIN can be broken into twelve
conceptsRecognizers
Interpreters Scenegraph
StrokeLibraries
Widgets
Views
Rendering
Transitions
Clipboard
Notification
Command
Events
![Page 21: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/21.jpg)
Nov 06 2000 21
Recognizers• Problem - many recognition techniques• Recognizers let new recognition
technologies be plugged into SATIN• Given a stroke, return n-best list• No actions taken
• SATIN comes with Rubine's Recognizer (implemented by Chris Long)
Recognizer0.860.650.63
CutCopyUndo
![Page 22: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/22.jpg)
Nov 06 2000 22
Interpreters• Problem – many ways of handling
strokes• Interpreters let new ways of handling
strokes be plugged into SATIN• e.g. straighten a line
• e.g. issue a command
Ink Interpreter
Gesture Interpreter erase me
![Page 23: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/23.jpg)
Nov 06 2000 23
Interpreters and Recognizers• Decoupled recognition (Recognizers)
from actions (Interpreters)• Interpreters can use recognizers
• Both are modules that can be plugged into apps built with SATIN
Interpreter Recognizer
Cut
0.860.650.63
CutCopyUndo
![Page 24: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/24.jpg)
Nov 06 2000 24
Multi-Interpreters• Problem – Hard to combine and extend
existing interpreters• Multi-Interpreters are a collection of
interpreters + dispatch policy– e.g. dispatch to chain of interpreters
– e.g. disable some interpreters on context
DefaultMulti Interpreter Intrp A Intrp Z…
Semantic ZoomMulti Interpreter
Intrp B
Intrp A Intrp Z…Intrp B
![Page 25: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/25.jpg)
Nov 06 2000 25
Phrase(Patch)
Scenegraph
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
…
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Phrase(Patch)
Timed Stroke
![Page 26: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/26.jpg)
Nov 06 2000 26
Stroke Dispatching• Needed an approach that enabled:
– Individual objects handle strokes on own– Priority to gestures over ink
• Default policy is top-down dispatch1. Process stroke with its gesture
interpreter2. Re-dispatch stroke to one of its children3. Process stroke with its ink interpreter4. Handle stroke in object itself
![Page 27: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/27.jpg)
Nov 06 2000 27
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
![Page 28: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/28.jpg)
Nov 06 2000 28
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
1. Process stroke with gesture interpreterSheet
DefaultMulti Interpreter
Hold Select Intrp
Standard Gesture Intrp…
![Page 29: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/29.jpg)
Nov 06 2000 29
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
![Page 30: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/30.jpg)
Nov 06 2000 30
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
Sheet
DefaultMulti Interpreter
Hold Select Intrp
Standard Gesture Intrp…
1. Process stroke with gesture interpreter
![Page 31: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/31.jpg)
Nov 06 2000 31
Stroke Dispatching2. Re-dispatch stroke to children
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
![Page 32: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/32.jpg)
Nov 06 2000 32
Stroke Dispatching1. Process stroke with gesture interpreter
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
![Page 33: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/33.jpg)
Nov 06 2000 33
Stroke Dispatching2. Re-dispatch stroke to children
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
![Page 34: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/34.jpg)
Nov 06 2000 34
Stroke Dispatching3. Process stroke with ink interpreter
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
Phrase Interpreter
![Page 35: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/35.jpg)
Nov 06 2000 35
Stroke Dispatching4. Handle stroke in object
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
Denim Sketch (Patch)
![Page 36: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/36.jpg)
Nov 06 2000 36
Stroke Dispatching
Phrase(Patch)
Sheet
Denim Label (Patch)Denim Sketch (Patch)
Timed Stroke
![Page 37: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/37.jpg)
Nov 06 2000 37
Example - Cut Shallow
![Page 38: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/38.jpg)
Nov 06 2000 38
Example - Cut Deep
![Page 39: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/39.jpg)
Nov 06 2000 39
Reusable Stroke LibrariesStraighten
Merge
![Page 40: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/40.jpg)
Nov 06 2000 40
Reusable Stroke LibrariesSplit
Simplify
![Page 41: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/41.jpg)
Nov 06 2000 41
Widgets • Pie Menus• Pen PLAF
• Swing "Pluggable Look And Feel"• Larger and clickable sliders
• Single-click file opener
![Page 42: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/42.jpg)
Nov 06 2000 42
Conclusions• SATIN - Java toolkit for informal ink
apps– Extensible thru Recognizers, Interpreters,
and Multi-Interpreters– Separates mechanism from policy for
strokes– Offers reusable libraries for manipulating
ink• Reusable widgets for pens
![Page 43: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/43.jpg)
Nov 06 2000 43
Download SATIN
http://guir.berkeley.edu/projects/satin
http://guir.berkeley.edu
Download SATIN at
![Page 44: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/44.jpg)
Nov 06 2000 44
Extra Slides
![Page 45: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/45.jpg)
Nov 06 2000 45
DENIM
![Page 46: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/46.jpg)
Nov 06 2000 46
DENIM
![Page 47: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/47.jpg)
Nov 06 2000 47
Survey of Informal Ink Apps• Elec. Cocktail
Napkin• Flatland• Pegasus• FiloChat• Zenpad• PerSketch• Teddy• PatchWork
• XLibris• NotePals• Dynomite• SILK• Tivoli• Kramer's
Patches• Music Notepad• Knight
![Page 48: Jason Hong and James Landay University of California Berkeley Group for User Interface Research](https://reader031.fdocuments.in/reader031/viewer/2022020219/56815bf2550346895dc9e24d/html5/thumbnails/48.jpg)
Nov 06 2000 48
Architecture Overview
• Recognizers• Interpreters• Scenegraph• Stroke Libraries• Widgets• Views
• Rendering• Transitions• Clipboard• Notifications• Commands• Events
SATIN can be broken into twelve concepts