Bricolage: Example-Based Retargeting for Web Design

Ranjitha Kumar Jerry O. Talton Salman Ahmad Scott R. KlemmerStanford University HCI Group

Department of Computer Science{ranju,jtalton,saahmad,srk}@cs.stanford.edu

ABSTRACTThe Web provides a corpus of design examples unparalleledin human history. However, leveraging existing designs toproduce new pages is often difficult. This paper introducesthe Bricolage algorithm for transferring design and contentbetween Web pages. Bricolage employs a novel, structured-prediction technique that learns to create coherent mappingsbetween pages by training on human-generated exemplars.The produced mappings are then used to automatically trans-fer the content from one page into the style and layout of an-other. We show that Bricolage can learn to accurately repro-duce human page mappings, and that it provides a general,efficient, and automatic technique for retargeting content be-tween a variety of real Web pages.

Author KeywordsWeb design, examples, retargeting, structured prediction.

ACM Classification KeywordsH.5.4 Information interfaces and presentation: Hypertext/Hypermedia - Theory.

General TermsAlgorithms, Design, Human Factors

INTRODUCTIONDesigners in many fields rely on examples for inspiration[17], and examples can facilitate better design work [22].Examples can illustrate the space of possible solutions andhow to implement those possibilities [2, 3]. Furthermore, re-purposing successful elements from prior ideas can be moreefficient than reinventing them from scratch [12, 21, 14].

The Web provides a corpus of design examples unparal-leled in human history: by 2008, Google had indexed morethan one trillion unique URLs [1]. However, we hy-pothesize that this rich resource is underutilized for designtasks. While current systems assist with browsing examplesand cloning individual design elements, adapting the gestaltstructure of Web designs remains a time-intensive, manualprocess [22, 10].

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHI 2011, May 7–12, 2011, Vancouver, British Columbia.Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00

Figure 1. Bricolage computes coherent mappings between Webpages by matching visually and semantically similar page elements.The produced mapping can then be used to guide the transfer ofcontent from one page into the design and layout of the other.

Most design reuse today is accomplished with templates[13]. Templates use standardized page semantics to ren-der content into predesigned layouts. This strength is also aweakness: templates homogenize page structure, limit cus-tomization and creativity, and yield cookie-cutter designs.Ideally, tools should offer both the ease of templates and thediversity of the entire Web. What if any Web page could bea design template?

This paper introduces the Bricolage algorithm for transfer-ring design and content between Web pages. The term“bricolage” refers to the creation of a work from a di-verse range of things that happen to be available. Bricolagematches visually and semantically similar elements in pagesto create coherent mappings between them. These mappingscan then be used to automatically transfer the content fromone page into the style and layout of the other (Figure 1).

Bricolage uses structured prediction [7] to learn how totransfer content between pages. It trains on a corpus ofhuman-generated mappings, collected using a Web-basedcrowdsourcing interface, the Bricolage Collector. The Col-lector was seeded with 50 popular Web pages that weredecomposed into a visual hierarchy by a novel, constraint-

based page segmentation algorithm, Bento. In an onlinestudy, 39 participants with some Web design experiencespecified correspondences between page regions and an-swered free-response questions about their rationale.

These mappings guided the design of Bricolage’s matchingalgorithm. We found consistent structural patterns in howpeople created mappings between pages. Participants notonly identified elements with similar visual and semanticproperties, but also used their location in the pages’ hier-archies to guide their assignments. Consequently, Bricolageemploys a novel tree-matching algorithm that flexibly bal-ances visual, semantic, and structural considerations. Wedemonstrate that this yields significantly more human-likemappings.

This paper presents the Bento page segmentation algorithm,the data collection study, the mapping algorithm, and the ma-chine learning method. It then shows results demonstratingthat Bricolage can learn to closely produce human mappings.Lastly, it illustrates how Bricolage is useful for a diverse setof design applications: for rapidly prototyping alternatives,retargeting content to alternate form factors such as mobiledevices, and measuring the similarity of Web designs.

THE BENTO PAGE SEGMENTATION ALGORITHMTo transfer content between Web pages, Bricolage first seg-ments each page into a hierarchy of salient regions that canbe extracted and manipulated. The page’s Document Ob-ject Model (DOM) tree, which describes the page’s content,structure, and style, provides a convenient starting point forthis segmentation [18].

Existing page segmentation algorithms begin by partitioningthe DOM into discrete, visually-salient regions [4, 5, 19].These algorithms produce good results whenever a page’sDOM closely mirrors its visual hierarchy, which is the casefor many simple Web pages.

However, these techniques fail on more complex pages.Modern CSS allows content to be arbitrarily repositioned,meaning that the structural hierarchy of the DOM may onlyloosely approximate the page’s visual layout. In our 50-pagecorpus, we found disparities between the DOM and the vi-sual layout an average of 2.3 times per page. Similarly, in-lined text blocks are not assigned individual DOM elements,and therefore cannot be separated from surrounding markup.In practice, these issues render existing segmentation algo-rithms poorly suited to real-world Web pages.

This paper introduces Bento, a page segmentation algorithmthat “re-DOMs” the input page in order to produce cleanand consistent segmentations. The algorithm comprises fourstages. First, each inlined element is identified and wrappedinside a <span> tag to ensure that all page content is con-tained within a leaf node of the DOM tree. Next, the hier-archy is reshuffled so that parent-child relationships in thetree correspond to visual containment on the page. EachDOM node is labeled with its rendered page coordinates andchecked to verify that its parent is the smallest region that

body

#header

#logo #nav

#content

h1.section

.col .col .col

#footer

Figure 2. Left: The colored boxes illustrate Bento’s segmentation.Right: Bento’s output tree and associated DOM nodes for this page.

contains it. When this constraint is violated, the DOM isadjusted accordingly, taking care to preserve layout detailswhen nodes are reshuffled. Third, redundant and superflu-ous nodes that do not contribute to the visual layout of thepage are removed. Fourth, the hierarchy is supplementedto introduce missing visual structures. These structural ele-ments are added by computing horizontal and vertical sepa-rators across each page region and inserting enclosing DOMnodes accordingly, similar to VIPS [4]. At the end of thesefour steps, all page content is assigned to a leaf node in theDOM tree, and every non-leaf node contains its children inscreen space (Figure 2).

Bento is available as a web service and a BSD open-sourceC++ library at http://hci.stanford.edu/bento.

COLLECTING AND ANALYZING HUMAN MAPPINGSRetargeting Web pages is closely related to automatic docu-ment layout and UI generation. In these domains, the state-of-the-art is constraint-based synthesis [18, 11], which be-gins with the designer building an abstract data model foreach individual class of designs. While this strategy workswell in highly-structured domains, the heterogeneous natureof the Web makes model construction impracticable.

We hypothesized that a more general retargeting schemecould be produced by training a machine learning algorithmon human-generated mappings between pages. To this end,we created the Bricolage Collector, a Web application forgathering human page mappings from online workers. Wedeployed the Collector online, and analyzed the producedmappings to understand how people map Web pages.

Study DesignWe selected a diverse corpus of 50 popular Web pages cho-sen from the Alexa Top 100, Webby award winners, highly-regarded design blogs, and personal bookmarks. Within thiscorpus, we selected a focus set of eight page pairs. Each par-ticipant was asked to match one or two pairs from the focusset, and one or two more chosen uniformly at random fromthe corpus as a whole. The Collector gathered data abouthow different people map the same pair of pages, and abouthow people map many different pairs. We recruited 39 par-ticipants through email lists and online advertisements. Eachreported some Web design experience.

Figure 3. The Bricolage Collector Web application asks users tomatch each highlighted region in the left (content) page to the corre-sponding region in the right (layout) page.

ProcedureParticipants watched a tutorial video demonstrating the Col-lector interface and describing the task. The video instructedparticipants to produce mappings for transferring the leftpage’s content into the right page’s layout. It emphasizedthat participants could use any criteria they deemed appro-priate to match elements. After the tutorial, the Collectorpresented participants with the first pair of pages (Figure 3).

The Collector interface iterates over the segmented regionsin the content page one at a time, asking participants to finda matching region in the layout page. The user selects amatching region via the mouse or keyboard, and confirmsit by clicking the MATCH button. If no good match existsfor a particular region, the user clicks the NO MATCH but-ton. After every fifth match, the interface presents a dialogbox asking,“Why did you choose this assignment?” Theserationale responses are logged along with the mappings, andsubmitted to a central server.

ResultsParticipants generated 117 mappings between 52 uniquepairs of pages: 73 mappings for the 8 pairs in the focusset, and 44 covering the rest of the corpus. They aver-aged 10.5 seconds finding a match for each page region(min = 4.42s, max = 25.0s), and 5.38 minutes per pagepair (min = 1.52m, max = 20.7m). Participants providedrationales for 227 individual region assignments, averaging4.7 words in length.

ConsistencyWe define the consistency of two mappings for the samepage pair as the percentage of page regions with identical as-signments. The average inter-mapping consistency of the fo-cus pairs was 78.3% (min = 58.8%, max = 89.8%). 37.8%of page regions were mapped identically by all participants.

RationaleParticipants provided rationales like “Title of rightmost bodypane in both pages.” We analyzed these rationales with La-tent Semantic Analysis (LSA), which extracts contextuallanguage usage in a set of documents [8]. LSA takes abag-of-words approach to textual analysis: each document istreated as an unordered collection of words, ignoring gram-

mar and punctuation. We followed the standard approach,treating each rationale as a document and forming the term-document matrix where each cell’s value counts the occur-rences of a term in a document. We used Euclidean normal-ization to make annotations of different lengths comparable,and inverse document-frequency weighting to deemphasizecommon words like a and the.

LSA decomposes the space of rationales into semantic “con-cepts.” Each concept is represented by a principal compo-nent of the term-document matrix, and the words with thelargest projections onto the component are the concept’s de-scriptors.

For the first component, the words with the largest projec-tions are: footer, link, menu, description, videos, picture, lo-gin, content, image, title, body, header, search, and graphic.These words pertain primarily to visual and semantic at-tributes of page content.

For the second component, the words with the largest pro-jections are: both, position, about, layout, bottom, one, two,three, subsection, leftmost, space, column, from, and hori-zontal. These words are mostly concerned with structuraland spatial relationships between page elements.

Structure and HierarchyTwo statistics examine the mappings’ structural and hierar-chical properties: one measuring how frequently the map-ping preserves ancestry, and the other measuring how fre-quently it preserves siblings.

We define two matched regions tobe ancestry preserving if their par-ent regions are also matched (see in-set). A mapping’s degree of ancestrypreservation is the number of ancestry-preserving regions divided by the total number of matchedregions. Participants’ mappings preserved ancestry 53.3%of the time (min = 7.6%, max = 95.5%).

Similarly, we define a set of page re-gions sharing a common parent to besibling preserving if the regions theyare matched to also share a commonparent (see inset). Participants pro-duced mappings that were 83.9% sibling preserving (min =58.3%, max = 100%).

COMPUTING PAGE MAPPINGSThe study’s results suggest that mappings produced by dif-ferent people are highly consistent: there is a “method to themadness” that may be learned. Moreover, the results sug-gest that algorithmically producing human-like mappingsrequires incorporating both semantic and structural con-straints, and learning how to balance between them.

Prior work in mapping HTML documents presents two dis-tinct approaches. The first ignores structural relationshipsbetween DOM nodes and maps elements irrespective of their

locations in the pages’ visual hierarchy [16]. The seconduses tree-matching techniques [20], which strictly preservehierarchical relationships: once two nodes have been placedin correspondence, their descendants must be matched aswell [28, 24, 27]. The results from our study suggest thatneither extreme is desirable.

Bricolage introduces a novel optimization algorithm whichflexibly balances semantic and structural constraints. The al-gorithm connects the nodes of the two page trees to form acomplete bipartite graph, and for each edge, assigns a costcomprising three terms. The first term measures visual andsemantic differences between the corresponding page ele-ments, the second penalizes edges that violate ancestry rela-tionships, and the third penalizes edges that break up siblinggroups. Determining the best page mapping then reduces tofinding a minimum-cost matching of the constructed graph.Bricolage uses structured prediction to learn a cost functionunder which the set of exemplar mappings are minimal [7].

T1 T2G

T1 T2M

Formally, given two page trees withnodes T1 and T2, we construct a com-plete bipartite graph G between T1 ∪{⊗1} and T2 ∪ {⊗2}, where ⊗1 and⊗2 are no-match nodes. These two no-match nodes enable the model to trackwhich nodes in one tree have no coun-terpart in the other. We then define apage mapping M to be a set of edgesfrom G such that every node in T1∪T2

is covered by precisely one edge. Inthis paper, given a tree node m, M(m)denotes its image (i.e., its counterpartin the other tree). The algorithm as-signs a cost c(e) to each edge e ∈ M ,and aggregates them to compute the to-tal mapping cost c(M) =

∑e∈M c(e).

Bricolage then searches for the least-cost mapping M? = argminM c(M).

Exact Edge CostsWe define the cost of an edge e ∈ T1 × T2 to be the sum ofthe visual, ancestry, and sibling costs

c(e) = cv(e) + ca(e) + cs(e).

For the edges inG connecting tree nodes to no-match nodes,we fix the cost c(e) = wn, where wn is a constant no-matchweight. The edge between the two no-match nodes is as-signed a cost of 0 to prevent it from influencing the finalmapping.

To compute cv([m,n]), the algorithm compares visual andsemantic properties of m and n by inspecting their DOMnodes. The Learning the Cost Model section describes thiscomputation in detail.

The ancestry cost ca(·) penalizes edges that violate ancestryrelationships between the pages’ elements. Consider a nodem ∈ T1, and let C(m) denote the children of m. We definethe ancestry-violating children of m, V (m), to be the set of

m n

n'

e

... ...... ... ...

Figure 4. To determine the ancestry penalty for an edge e = [m,n],Bricolage counts the children of m and n which induce ancestry vio-lations. In this example, n′ is an ancestry-violating child of n becauseit is not mapped to a child of m; therefore, n′ induces an ancestrycost on e.

m’s children that map to nodes that are notM(m)’s children,i.e.,

V (m) = {m′ ∈ C(m) |M(m′) ∈ T2 r C(M(m))} ,and define V (n) symmetrically. Then, the ancestry cost foran edge is proportional to the number of ancestry violatingchildren of its terminal nodes

ca([m,n]) = wa (|V (m)|+ |V (n)|) ,where wa is a constant ancestry violation weight (see Fig-ure 4).

The sibling cost cs(·) penalizes edges that fail to preservesibling relationships between trees. To calculate this term,we first define a few tree-related concepts. Let P (m)denote the parent of m. Then, the sibling group of anode m is the set comprising the children of its parent:S(m) = {C(P (m))}. Given a mapping M , the sibling-invariant subset of m, I(m), is the set of nodes in m’s sib-ling group that map to nodes in M(m)’s sibling group, i.e.,

I(m) = {m′ ∈ S(m) |M(m′) ∈ S(M(m))} ;

the sibling-divergent subset of m, D(m), is the set of nodesin m’s sibling group that map to nodes in T2 not in M(m)’ssibling group, i.e.,

D(m) = {m′ ∈ S(m) r I(m) |M(m′) ∈ T2} ;

and the set of distinct sibling families that m’s sibling groupmaps into is

F (m) =⋃

m′∈S(m)

P (M(m′)).

We define all corresponding terms for n symmetrically, andthen compute the total sibling cost:

cs([m,n]) = ws

(|D(m)|

|I(m)||F (m)|+

|D(n)||I(n)||F (n)|

),

where ws is a constant sibling violation weight. The tworatios increase when siblings are broken up (i.e., their imageshave different parents), and decrease when more siblings arekept together (see Figure 5).

n n' n''m

... ...... ... ...

e

Figure 5. To determine the sibling penalty for an edge e = [m,n],Bricolage computes the sibling-invariant and sibling-divergent sub-sets of m and n. In this example, I(n) = {n′} and D(n) = {n′′};therefore, n′ decreases the sibling cost on e and n′′ increases it.

Bounding Edge CostsWhile this cost model balances semantic, ancestral, and sib-ling constraints, it cannot be used to search for the optimalmapping M? directly. Although cv([m,n]) can be evaluatedfor an edge by inspecting m and n, ca(·) and cs(·) requireinformation about the other edges in the mapping.

While we cannot precisely evaluate ca(·) and cs(·) a priori,we can compute bounds for them on a per-edge basis [6].Moreover, each time we accept an edge [m,n] into M , wecan remove all the other edges incident on m and n fromG. Each time we prune an edge in this way, the bounds forother nearby edges may be improved. Therefore, we employa Monte Carlo algorithm to approximate M?, stochasticallyfixing an edge in G, pruning away the other edges incidenton its nodes, and updating the bounds on those that remain.

To bound the ancestry cost of an edge [m,n] ∈ G, we mustconsider each child of m and n and answer two questions.First, is it impossible for this node to induce an ancestry vi-olation? Second, is it unavoidable that this node will inducean ancestry violation? The answer to the first question in-forms the upper bound for ca(·); the answer to the secondinforms the lower.

A nodem′ ∈ C(m) can induce an ancestry violation if thereis some edge between it and a node in T2 r (C(n)∪ {⊗2}).Conversely, m′ is not guaranteed to induce an ancestryviolation if some edge exists between it and a node inC(n) ∪ {⊗2} . Accordingly, we define indicator functions

1Ua (m′, n) =

{1 if ∃[m′, n′] ∈ G s.t. n′ 6∈ C(n) ∪ {⊗2}0 else

,

1La (m′, n) =

{1 if 6 ∃[m′, n′] ∈ G s.t. n′ ∈ C(n) ∪ {⊗2}0 else

.

Then, the upper and lower bounds for ca([m,n]) are

Ua([m,n]) =

wa

∑m′∈C(m)

1Ua (m′, n) +∑

n′∈C(n)

1Ua (n′,m)

,

m n

n' n''m' m''

e

... ... ... ...

Figure 6. To bound ca([m,n]), observe that neither m′ nor n′ caninduce an ancestry violation. Conversely, m′′ is guaranteed to vi-olate ancestry. No guarantee can be made for n′′. Therefore, thelower bound for ca is wa, and the upper bound is 2wa.

and

La([m,n]) =

wa

∑m′∈C(m)

1La (m′, n) +∑

n′∈C(n)

1La (n′,m)

.

Figure 6 illustrates the computation of these bounds. Prun-ing edges from G causes the upper bound for ca([m,n]) todecrease, and the lower bound to increase.

Similarly, we can bound cs([m,n]) by bounding the numberof divergent siblings, invariant siblings, and distinct families:|D(·)|, |I(·)|, and |F (·)|. Let S̄(m) = S(m) r {m} andconsider a node m′ ∈ S̄(m). It is possible that m′ is inD(m) as long as some edge exists between it and a node inT2 r (S̄(n) ∪ {⊗2}). Conversely, m′ cannot be guaranteedto be in D(m) as long as some edge exists between it and anode in S̄(n) ∪ {⊗2}. Then, we have

1UD(m′, n) =

{1 if ∃[m′, n′] ∈ G s.t. n′ 6∈ S̄(n) ∪ {⊗2}0 else

,

UD(m,n) =∑

m′∈S̄(m)

1UD(m′, n),

and

1LD(m′, n) =

{1 if 6 ∃[m′, n′] ∈ G s.t. n′ ∈ S̄(n) ∪ {⊗2}0 else

,

LD(m,n) =∑

m′∈S̄(m)

1LD(m′, n).

The bounds for |I(m)| are similarly given by

1UI (m′, n) =

{1 if ∃[m′, n′] ∈ G s.t. n′ ∈ S̄(n)0 else

,

UI(m,n) = 1 +∑

m′∈S̄(m)

1UI (m′, n),

n n''m

e

... ... ...

m'

...

m''

...

n'

...

Figure 7. To bound cs([m,n]), observe that m′ is guaranteed to bein I(m), and m′′ is guaranteed to be in D(m). No guarantees canbe made for n′ and n′′. Therefore, the lower bound for cs is ws/4,and the upper bound is 3ws/4.

and

1LI (m′, n) =

{1 if ∀[m′, n′] ∈ G, n′ ∈ S̄(n)0 else

,

LI(m,n) = 1 +∑

m′∈S̄(m)

1LI (m′, n).

For all nonzero sibling costs, the lower bound for |F (m)|is 2 and the upper bound is LD(m,n) + 1. All remainingquantities are defined symmetrically. Then, upper and lowerbounds for cs([m,n]) are given by

Us([m,n]) =ws

2

(UD(m,n)

LI(m,n)+UD(n,m)

LI(n,m)

)and

Ls([m,n]) =

ws

(LD(m,n)

UI(m,n) (LD(m,n) + 1)+

LD(n,m)

UI(n,m) (LD(n,m) + 1)

).

Figure 7 illustrates these computations.

With bounds for the ancestry and sibling terms in place, up-per and lower bounds for the total edge cost may be triv-ially computed as cU (e) = cv(e) + Ua(e) + Us(e) andcL(e) = cv(e) + La(e) + Ls(e).

Approximating the Optimal MappingTo approximate the optimal mappingM∗, we use the Metropo-lis algorithm [23]. We represent each matching as an orderedlist of edges M , and define a Boltzmann-like objective func-tion

f(M) = exp [−β c(M)] ,

where β is a constant. At each iteration of the algorithm, anew mapping M̂ is proposed, and becomes the new refer-ence mapping with probability

α(M̂ |M) = min

(1,f(M̂)

f(M)

).

The algorithm runs for N iterations, and the mapping withthe lowest cost is returned.

To initialize M , the bipartite graph G is constructed and theedge bounds initialized. Then, the edges in G are traversedin order of increasing bound. Each edge is considered for as-signment to M with some fixed probability γ, until an edgeis chosen. If the candidate edge can be fixed and at leastone complete matching still exists, it is appended to M , theother edges incident on its terminal nodes are pruned, andthe bounds for the remaining edges in G are tightened.

To propose M̂ , we choose a random index j ∈ [1, |M |].Then, we re-initialize G, and fix the first j edges in M . Toproduce the rest of the matching, we repeat the iterative edgeselection process described above. In our implementation,we take γ = .7 and N = 100; β is chosen on a per-domainbasis, based on the size of the trees.

LEARNING THE COST MODELWhile this mapping algorithm can be used with any visualand semantic cost model and weightswn,wa, andws, Brico-lage seeks to learn a model that will produce human-likemappings. It employs a feature-based approach to computethe visual and semantic cost cv(·) between nodes, and trainsthe weights of these features and those for the no-match, an-cestry, and sibling terms.

Edge FeaturesThe algorithm computes a set of visual and semantic proper-ties for each node in the page trees. Visual properties arecomputed using a node’s render-time appearance, and in-clude attributes like width, font size, and mean RGB val-ues. Semantic properties take Boolean values, computed byattribute tests such as “is an image” or “is contained in theheader.” The Appendix gives a full list of these properties.

To compute an edge’s total cost, the algorithm first calcu-lates the difference between each property for m and n, andconcatenates these values—along with the exact ancestryand sibling costs and a Boolean no-match indicator—intoa feature vector fe. Given a set of weights wf for each vi-sual and semantic feature, the edge cost is then computed asc(e) = w̄T fe, where w̄ = 〈wf , wa, ws, wn〉.

Given a mapping M , the algorithm assembles an aggregatefeature vector FM =

∑e∈M fe to calculate c(M) = w̄TFM .

Training the cost model then reduces to finding a set ofweights under which the mappings in the training set haveminimal total cost.

Generalized Perceptron AlgorithmTo learn a consistent assignment for the weight vector w̄ un-der which the set of exemplar mappings are minimal, Brico-lage uses the generalized perceptron algorithm for structuredprediction [7].

The perceptron begins by initializing w̄0 = 0. In each sub-sequent iteration, the perceptron randomly selects a pairof page trees and an associated human mapping M fromthe training set. Next, using the current weight vectorw̄i, it computes a new mapping M̂ ≈ argminM w̄T

i FM .Based on the resultant mapping, a new aggregate feature

Source Target Synthesized

Figure 8. A current limitation of the content transfer algorithm il-lustrating the challenges of HTML/CSS. The target page’s CSS pre-vents the bounding beige box from expanding. This causes the textto overflow (synthesized page). Also, the target page expects allheaders to be images. This causes the “About Me” header to dis-appear (synthesized page). An improved content transfer algorithmcould likely address both of these issues.

vector FM̂ is calculated, and the weights are updated byw̄i+1 = w̄i + αi

(FM̂ − FM

), where αi = 1/

√i+ 1 is the

learning rate.

The perceptron algorithm is only guaranteed to converge ifthe training set is linearly separable; in practice, it producesgood results for many diverse data sets [7]. Since the weightsmay oscillate during the final stages of the learning, the fi-nal cost model is produced by averaging over the last fewiterations.

CONTENT TRANSFEROnce a cost model is trained, it is fed to the matching algo-rithm, which uses it to predict mappings between any twopages. Bricolage then uses these computed mappings to au-tomatically transfer the content from one page into the styleand layout of another. In its segmented page representation,page content (text, images, links, form fields) lives on theleaf nodes of the page tree. Before transferring content, theinner HTML of each node in the source page is preprocessedto inline CSS styles and convert embedded URLs to absolutepaths. Then, content is moved between mapped nodes by re-placing the inner HTML of the target node with the innerHTML of the source node.

Content matched to a no-match node can be handled in oneof two ways. In the simplest case, unmatched source nodesare ignored. However, if important content in the sourcepage is not mapped, it may be more desirable to insert theunmatched node into the target page parallel to its mappedsiblings, or beneath its lowest mapped ancestor.

This approach works well for many pages. Occasionally,the complexity and diversity of modern Web technologiespose practical challenges to resynthesizing coherent HTML.Many pages specify style rules and expect certain markuppatterns, which may cause the new content to be renderedincorrectly (Figure 8). Furthermore, images and plugin ob-jects (e.g., Flash, Silverlight) have no CSS style informa-tion that can be borrowed; when replaced, the new contentwill not exhibit the same visual appearance and may seemout of place. Lastly, embedded scripts are often tightly cou-pled with the original page’s markup and break when naı̈velytransferred. Consequently, the current implementation ig-nores them, preventing dynamic behavior from being bor-rowed. A more robust content transfer algorithm is requiredto address these issues and remains future work.

RESULTSWe demonstrate the efficacy of Bricolage in two ways. First,we show several practical examples of Bricolage in action.Second, we evaluate the machine learning components byperforming a hold-out cross-validation experiment on thegathered human mappings.

ExamplesFigure 10 demonstrates the algorithm in a rapid prototyp-ing scenario, in which an existing page is transformed intoseveral potential replacement designs. Creating multiple al-ternatives facilitates comparison, team discussion, and de-sign space exploration [9, 15, 25]. Figure 11 demonstratesthat Bricolage can be used to retarget content across formfactors, showing a full-size Web page automatically mappedinto two different mobile layouts.

Figure 9 illustrates an ancillary benefit of Bricolage’s costmodel. Since Bricolage searches for the optimal mappingbetween pages, the returned cost can be interpreted as an ap-proximate distance metric on the space of page designs. Al-though the theoretical properties of this metric are not strong(it satisfies neither the triangle inequality nor the identity ofindiscernibles), in practice it may provide a useful mecha-nism for automatically differentiating between pages withsimilar and dissimilar designs.

Machine Learning ResultsTo test the effectiveness of Bricolage’s machine learningcomponents, we ran a hold-out test. We used the 44 col-lected mappings outside the focus set as training data, and

Most DissimilarMost Similar

Figure 9. Bricolage can be used to induce a distance metric on the space of Web designs. By mapping the leftmost page onto each of thepages in the corpus and examining the mapping cost, we can automatically differentiate between pages with similar and dissimilar designs.

Figure 10. Bricolage used to rapidly prototype many alternatives. Top-left: the original Web page. Rest: the page automatically retargeted tothree other layouts and styles.

Figure 11. Bricolage can retarget Web pages designed for the desktop to mobile devices. Left: the original Web page. Right: the pageautomatically retargeted to two different mobile layouts.

the mappings in the focus set as test data. The perceptronwas run for 400 iterations, and the weight vector averagedover the last 20. The learned cost model was used to predictmappings for each of the 8 focus pairs. Table 1 shows thecomparison between the learned and reference mappings us-ing three different metrics: average similarity, nearest neigh-bor similarity, and percentage of edges that appear in at leastone mapping.

The online mapping experiment found a 78% inter-mappingconsistency between the participants. This might be con-sidered a gold standard against which page mapping algo-rithms are measured. Currently, Bricolage achieves a 69%consistency. By this measure, there is room for improve-ment. However, Bricolage’s mappings overlap an averageof 78% with their nearest human neighbor, and 88% of theedges generated by Bricolage appear in some human map-ping.

This structured prediction approach was motivated by thehypothesis that ancestry and sibling relationships are crucialto predicting human mappings. We tested this hypothesisby training three additional cost models containing differentfeature subsets: visual terms only, visual and ancestry terms,and visual and sibling terms. Considering only local featuresyields an average nearest neighbor match of 53%; mappingwith local and sibling features yields 67%; mapping withlocal and ancestry features yields 75%. Accounting for allof these features yields 78%, a result that dominates that ofany subset. In short, flexibly preserving structure is crucialto producing good mappings.

IMPLEMENTATIONBricolage’s page segmentation, mapping, and machine learn-ing libraries are implemented in C++ using the Qt frame-work, and use Qt’s WebKit API in order to interface directlywith a browser engine. Once a cost model has been trained,Bricolage produces mappings between pages in about 1.04seconds on a 2.55 Ghz Intel Core i7, averaging roughly 0.02seconds per node.

The corpus pages are archived using the Mozilla ArchiveFile Format and hosted on a server running Apache. Forefficiency, page segmentations and associated DOM nodefeatures are computed and cached for each page when itis added to the corpus. Each feature has its own dynamicplug-in library, allowing the set of features to be extendedwith minimal overhead, and mixed and matched at runtime.The Bricolage Collector is written in HTML, Javascript, andCSS. Mapping results are sent to a centralized Ruby on Railsserver and stored in a SQLite database.

CONCLUSIONS AND FUTURE WORKThis paper introduced the Bricolage algorithm for automat-ically transferring design and content between Web pages.Bricolage’s major algorithmic insight was a technique forcapturing the structural relationships between elements, andusing an optimization approach to balance local and globalconcerns. This work takes a first step towards a powerfulnew paradigm for example-based Web design, and opens upexciting areas for future research.

Table 1. Results of the hold-out cross-validation experiment. Brico-lage performs substantially worse without both the ancestry and sib-ling terms in the cost model.

The current prototype employs thirty visual and semanticfeatures. Adding more sophisticated properties—such asthose based on computer vision techniques—will likely im-prove the quality of the machine learning.

Future work could extend example-based design to other do-mains. The current Bricolage implementation is HTML spe-cific. In principle, the retargeting algorithm can be appliedto any document with hierarchical structure such as slidepresentation and vector graphics files. With richer visiontechniques (along the lines of [26]), the Bricolage approachmight extend to documents and interfaces without accessiblestructure.

Finally, an important next step is to create a retargeting de-sign tool that allows both novice and experts to more cre-atively use examples. Observing how people use such a toolwill provide valuable research knowledge about the role ex-amples can play in amplifying creativity.

ACKNOWLEDGEMENTSWe thank Tim Roughgarden for helpful discussions aboutthe matching algorithm; Chuong Do for helpful discussionsabout structured prediction techniques; Amy Jang for im-plementing machine learning features; Juho Kim and KanyaSiangliulue for helping prototype the Bricolage Collector;Mira Dontcheva and the reviewers for their helpful com-ments. This research was supported in part by NSF GrantIIS-0745320.

REFERENCES1. Alpert, J., Hajaj, N. We knew the web was big...

[online]. 2008. Available from:http://goo.gl/RtmG [cited 14 January 2011].

2. Brandt, J., Guo, P. J., Lewenstein, J., Dontcheva, M.,Klemmer, S. R. Two studies of opportunisticprogramming: interleaving Web foraging, learning, andwriting code. Proc. CHI (2009), ACM, pp. 1589–1598.

3. Buxton, B. Sketching User Experiences. MorganKaufmann, 2007.

4. Cai, D., Yu, S., Wen, J.-R., Ma, W.-Y. VIPS: avision-based page segmentation algorithm. Tech. Rep.MSR-TR-2003-79, Microsoft, 2003.

5. Chakrabarti, D., Kumar, R., Punera, K. Agraph-theoretic approach to Webpage segmentation.Proc. WWW (2008), pp. 377–386.

6. Chawathe, S. S., Garcia-Molina, H. Meaningful changedetection in structured data. Proc. SIGMOD (1997),ACM, pp. 26–37.

7. Collins, M. Discriminative training methods for hiddenMarkov models: theory and experiments withperceptron algorithms. Proc. EMNLP (2002), ACL.

8. Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer,T. K., Harshman, R. Indexing by latent semanticanalysis. Journal of the American Society forInformation Science 41, 6 (1990), pp. 391–407.

9. Dow, S. P., Glassco, A., Kass, J., Schwarz, M.,Schwartz, D. L., Klemmer, S. R. Parallel prototypingleads to better design results, more divergence, andincreased self-efficacy. ACM TOCHI 17(4) (December2010), pp. 1–24.

10. Fitzgerald, M. CopyStyler: Web design by example.Tech. rep., MIT, May 2008.

11. Gajos, K., Weld, D. S. SUPPLE: automaticallygenerating user interfaces. Proc. IUI (2004), ACM.

12. Gentner, D., Holyoak, K., Kokinov, B. The AnalogicalMind: Perspectives From Cognitive Science. MITPress, 2001.

13. Gibson, D., Punera, K., Tomkins, A. The volume andevolution of Web page templates. Proc. WWW (2005),ACM, pp. 830–839.

14. Hartmann, B., Wu, L., Collins, K., Klemmer, S. R.Programming by a sample: rapidly creating Webapplications with d.mix. Proc. UIST (2007), ACM.

15. Hartmann, B., Yu, L., Allison, A., Yang, Y., Klemmer,S. R. Design as exploration: creating interfacealternatives through parallel authoring and runtimetuning. Proc. UIST (2008), ACM.

16. Hashimoto, Y., Igarashi, T. Retrieving web pagelayouts using sketches to support example-based webdesign. 2nd Eurographics Workshop on Sketch-BasedInterfaces and Modeling (2005).

17. Herring, S. R., Chang, C.-C., Krantzler, J., Bailey, B. P.Getting inspired!: understanding how and whyexamples are used in creative design practice. Proc.CHI (2009), ACM, pp. 87–96.

18. Hurst, N., Li, W., Marriott, K. Review of automaticdocument formatting. Proc. DocEng (2009), ACM.

19. Kang, J., Yang, J., Choi, J. Repetition-based Web pagesegmentation by detecting tag patterns for small-screen

devices. IEEE Transactions on Consumer Electronics56 (May 2010), pp. 980–986.

20. Kim, Y., Park, J., Kim, T., Choi, J. Web informationextraction by html tree edit distance matching.International Conference on Convergence InformationTechnology. (2007), pp. 2455–2460.

21. Kolodner, J. L., Wills, L. M. Case-based creativedesign. In AAAI Spring Symposium on AI andCreativity (1993), pp. 50–57.

22. Lee, B., Srivastava, S., Kumar, R., Brafman, R.,Klemmer, S. R. Designing with interactive examplegalleries. Proc. CHI (2010), ACM.

23. Press, W. H., Teukolsky, S. A., Vetterling, W. T.,Flannery, B. P. Numerical Recipes: The Art of ScientificComputing, 3rd ed. Cambridge University Press, NewYork, NY, USA, 2007.

24. Shasha, D., Wang, J. T.-L., Zhang, K., Shih, F. Y. Exactand approximate algorithms for unordered treematching. IEEE Transactions on Systems, Man, andCybernetics 24, 4 (1994), pp. 668–678.

25. Talton, J. O., Gibson, D., Yang, L., Hanrahan, P.,Koltun, V. Exploratory modeling with collaborativedesign spaces. Proc. SIGGRAPH ASIA (2009), ACMPress.

26. Wattenberg, M., Fisher, D. A model of multi-scaleperceptual organization in information graphics. Proc.INFOVIS (2003), pp. 23 –30.

27. Zhang, K. A constrained edit distance betweenunordered labeled trees. Algorithmica 15, 3 (1996).

28. Zhang, K., Shasha, D. Simple fast algorithms for theediting distance between trees and related problems.SIAM J. Comput. 18, 6 (1989), pp. 1245–1262.

APPENDIXThe Bricolage prototype uses the following DOM propertiesas features in the learning.

The visual properties include: width, height, area, aspectRa-tio, fontSize, fontWeight, meanColor, numLinks, numCol-ors, numChildren, numImages, numSiblings, siblingOrder,textArea, wordCount, treeLevel, verticalSidedness (normal-ized distance from the horizon of the page), horizontalSid-edness (normalized distance from the midline of the page),leftSidedness (normalized distance from the left border ofthe page), topSidedness (normalized distance from the topborder of the page), and shapeAppearance (the minimum ofthe aspect ratio and its inverse).

The semantic properties include: search, footer, header, im-age, logo, navigation, bottom (if the node is in the bottom10% of the page), top (if the node is in the top 10% of thepage), fillsHeight (if the node extends more than 90% downthe page), and fillsWidth (if the node extends more than 90%across the page).