Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web...
-
Upload
liam-turrell -
Category
Documents
-
view
216 -
download
2
Transcript of Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web...
![Page 1: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/1.jpg)
stanford hci group
http://hci.stanford.eduUIST · 10 October 2007
Programming by a Sample: Rapidly Creating Web Applications with d.mix
Björn Hartmann, Leslie WuKevin Collins, Scott R. Klemmer
![Page 2: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/2.jpg)
2
How would you share UIST highlights with colleagues back home?
![Page 3: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/3.jpg)
3
How would you retrieve the same data programmatically?
It’s easy to understand the sites, but not the services.
![Page 4: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/4.jpg)
4
![Page 5: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/5.jpg)
5
Web sites and their APIs are correlated…
[flickr.com]
…let’s leverage that fact!
≈
![Page 6: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/6.jpg)
6
Give me the code for this!
![Page 7: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/7.jpg)
7
To retrieve this image, use:flickr.photos.getInfo(
user_id = '73866493@N00', photo_id= ‘3208312’)
![Page 8: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/8.jpg)
8
d.mix active wikiSource code generated by d.mix
Rendered Page
is executed in the active wiki
![Page 9: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/9.jpg)
9
Scenario
![Page 10: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/10.jpg)
10
![Page 11: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/11.jpg)
11
![Page 12: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/12.jpg)
12
![Page 13: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/13.jpg)
13
d.mix users
Lead Users Web Developers End Users
Site owners or lead users define mappings between sites and services (once).
![Page 14: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/14.jpg)
14
Lead Users Web Developers End Users
Web developers create d.mix applications
![Page 15: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/15.jpg)
15
Lead Users Web Developers End Users
End-users run (and tailor) applicationsin the d.mix wiki.
![Page 16: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/16.jpg)
16
d.mix Proxy Architecture
Original PageRewritten page with API annotationsProxy Server
Site-to-Service Map(hosted on d.mix wiki)
![Page 17: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/17.jpg)
17
Authoring the Site-to-Service Map…Without Help From the Site Owner
1 Map URL to Page Type2 Identify visual elements in page to
annotate (using XPath/CSS selectors)
3 Extract arguments for service calls from page source
4 Bind arguments to web service code snippet
![Page 18: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/18.jpg)
18
Why Not Just Scrape?
• Scraping at design-time rather than at run-time minimizes brittleness
• Web service calls can be parameterized
• Scraping at run-time can lead to lock-out
![Page 19: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/19.jpg)
19
![Page 20: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/20.jpg)
20
Page URL:flickr.com/photos/<username>/<photoid>/…
Regular Expression:%r{flickr.com//?photos/ [^/]+/\d+/?&script}
![Page 21: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/21.jpg)
21
Photo Title
Image URL
Tag Search
![Page 22: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/22.jpg)
22
Photo Title
Image URL
Tag Search
(doc/"#title_div")
(doc/"div.photoImgDiv")
(doc/"div.TagList")
![Page 23: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/23.jpg)
23
flickr.photos.getInfo( photo_id = “298655528”).title
info = flickr.photos.getInfo( photo_id = “298655528”)URL = “http://farm”+ info.farm-id + “.static.flickr.com/” + info.server-id + “/” + info.attributes[”id”] + “_” + info.secret + “.jpg”
flickr.photos.search( tags = “yosemite ...”)
Extracted from page source:Within <div> for all tags:
tag=div.at("a.Plain").inner_html
![Page 24: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/24.jpg)
24
Putting it all together…
def self.annotate_photopage_tags(doc)(doc/"div").reject{…}.each do |div|
tag = div.at("a.Plain").inner_htmlsrc = generate_tag_search_source (...)doc.at("body").inner_html +=
make_context_menu(div.at("a.Plain"),["Images matching tag #{tag}"],[src])
end
![Page 25: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/25.jpg)
25
Putting it all together…
Extract tag namedef self.annotate_photopage_tags(doc)(doc/"div").reject{…}.each do |div|
tag = div.at("a.Plain").inner_htmlsrc = generate_tag_search_source (...)doc.at("body").inner_html +=
make_context_menu(div.at("a.Plain"),["Images matching tag #{tag}"],[src])
end
![Page 26: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/26.jpg)
26
def self.annotate_photopage_tags(doc)(doc/"div").reject{…}.each do |div|
tag = div.at("a.Plain").inner_htmlsrc = generate_tag_search_source (...)doc.at("body").inner_html +=
make_context_menu(div.at("a.Plain"),["Images matching tag #{tag}"],[src])
end
Putting it all together…
Instantiate Source Example
![Page 27: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/27.jpg)
27
Putting it all together…
Add annotation to original page
def self.annotate_photopage_tags(doc)(doc/"div").reject{…}.each do |div|
tag = div.at("a.Plain").inner_htmlsrc = generate_tag_search_source (...)doc.at("body").inner_html +=
make_context_menu(div.at("a.Plain"),["Images matching tag #{tag}"],[src])
end
![Page 28: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/28.jpg)
28
Dataflow Summary
d.mix run time
d.mix design time
invokes
Web service
sends
code to
Web site
addsannotation
![Page 29: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/29.jpg)
29
Active Wiki• Wiki editor provides
syntax-highlighting for Ruby script
• Sandboxed execution runs script with limited capabilities
• Libraries facilitate invoking web services and manipulating results
![Page 30: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/30.jpg)
30
Beyond the Desktop Browser
![Page 31: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/31.jpg)
31
What we borrowed and what we wrote
Front end•Yahoo! UI (annotation UI)•JQuery (selectors)•Bookmarklet•UI logic
Back end•MouseHole (proxy)•Tepee (wiki)•Freakyfreakysandbox (safe execution)•Hpricot (selectors)•Annotation code•Site-to-service map•API wrappers•d.mix library
![Page 32: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/32.jpg)
32
Prototype Site-to-Service LibraryAPI Supported Actions Site-to-
Service map code size
• Get images from a user’s photo stream, with or without meta data• Get images from an image set• Get images from individual photo pages• Get images matching tags, global or per user, from tag clouds and photo pages• Get images by global image id• Get images from full-text search
355 lines
• Single web search result• Web search result set
54 lines• Retrieve a user’s videos• Retrieve most recent videos• Single search result• Search result set
115 lines
![Page 33: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/33.jpg)
33
First-use Lab Study (n=8)
• All participants had some programming experience, knew HTML
• Four had no experience with web APIs
• 75 minute sessions:Demonstration, warm-up, two design tasks
![Page 34: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/34.jpg)
34
Total lines of code
54
Written by participant
13
Generated by d.mix
41
Generated lines modified by participant
4
![Page 35: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/35.jpg)
35
![Page 36: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/36.jpg)
36
Lessons LearnedHow do I know what I can
sample?
?
![Page 37: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/37.jpg)
37
Lessons LearnedHow do I know what I can
sample?
![Page 38: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/38.jpg)
38
Sample from link to content
Sample content directly
Lessons LearnedOffer multiple ways to sample
information.
![Page 39: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/39.jpg)
39
LimitationsProxying the logged-in web is
challenging
![Page 40: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/40.jpg)
40
LimitationsProxying the logged-in web is
challenging
![Page 41: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/41.jpg)
41
LimitationsHow can one sample APIs that
provide interactive widgets intead of data?
![Page 42: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/42.jpg)
42
Related Work
Greasemonkey
Chickenfoot [Bolin, UIST2005]
Koala [Little, CHI2007]
Yahoo! Pipes
Open Kapow
Marmite [Wong, CHI2007]
IBM QEDWiki
Intel MashMaker [Ennals, SIGMOD2007]
Relations, Cards, and Search Templates [Dontcheva, UIST2007]
Citrine [Stylos, UIST2004]
WinCuts[Tan, CHI2004]
Clip, connect, clone[Fujima, UIST2004]
Hunter Gatherer[schraefel, WWW2002]
Facades[Stuerzlinger, UIST2006]
End-user Page Modification &Automation
End-user PAGECreation
Deep Copy & paste
Mica[Stylos, VL/HCC2006]
Assieme[Hoffmann, UIST2007]
Finding APIExamples
![Page 43: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/43.jpg)
43
Greasemonkey
Chickenfoot [Bolin, UIST2005]
Koala [Little, CHI2007]
End-user Page Modification &Automation
![Page 44: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/44.jpg)
44
Yahoo! Pipes
Open Kapow
Marmite [Wong, CHI2007]
IBM QEDWiki
Intel MashMaker [Ennals, SIGMOD2007]
Relations, Cards, and
Search Templates [Dontcheva, UIST2007]
End-user AuTHORING TOOLS
![Page 45: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/45.jpg)
45
Assieme[Hoffmann, UIST2007]
Mica[Stylos, VL/HCC2006]
API SEARCHTOOLS
![Page 46: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/46.jpg)
46
Contributions
• Search for programming examples in the solution domain, not the code domain.
• d.mix instantiates this idea for web service APIs through a site-to-service map.
• Integration of page annotation and script hosting enables rapid experimentation.
![Page 47: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/47.jpg)
47
Current WorkRe:MixReformatting existing web applications formobile device use
JuxtaposeExploring designalternatives inparallel
![Page 48: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/48.jpg)
48
AcknowledgmentsFunding
NSF grant IIS-0534662SAP Stanford Graduate Fellowship
Microsoft New Faculty FellowshipIntel (equipment donation)
Help Wendy Ju, Leith Abdulla, Michel
Krieger,whytheluckystiff
Imagesmorguefile.com
![Page 49: Stanford hci group UIST · 10 October 2007 Programming by a Sample: Rapidly Creating Web Applications with d.mix Björn Hartmann,](https://reader036.fdocuments.in/reader036/viewer/2022062712/56649c9b5503460f949596bc/html5/thumbnails/49.jpg)
stanford hci group
hci.stanford.edu/dmix