Dallas, August 18-22 Volume 20, Number 4, 1986 II I I I I I

H a i r y B r u s h e s

b y S t e v e S t r a s s m a n n C o m p u t e r G r a p h i c s and A n i m a t i o n G r o u p

M I T Med ia L a b o r a t o r y C a m b r i d g e , M a s s a c h u s e t t s

Arpanet: straz@med:ia-lab, mi't, edu

.Abs t r ac t

Paint brushes are modeled as a collection of bris- tles which evolve over the course of the stroke, leaving a realistic image of a sumi brush stroke. The major representational units are {1) Brush: a compound ob- ject composed of bristles, (2} Stroke: a trajectory of position and pressure, {3} Dip: a description of the application of paint to a class of brushes, and (4 / Pa- per: a mapping onto the display device. This modular system allows experimentation with various stochastic models of irk flow and color change. By selecting from a library of brushes, dips, and papers, the stroke can take on a wide variety of expressive textures.

1 Introduction

T h e "brushes" used in conven t iona l c o m p u t e r pa i n t i ng sys- t e m s are far s impler t h a n real pa in t b rushes . Usual ly no more t h a n a u t o m a t e d rubber s t a m p s , they bui ld up images by placing repea ted copies of some s t a t i c or s imply der ived p a t t e r n . Some s y s t e m s offer "a i rb rushes , " which s i m u l a t e a sp ray of ink by p a i n t i n g pixels in a c i rcular region a round the b rush .

T h i s pape r descr ibes an inves t iga t ion in to a far more re- alist ic mode l of pa in t ing . T h e image left by a sopp ing we t b ru sh or c rumb ly c rayon d ragged er ra t ica l ly across a shee t of t e x t u r e d pape r can be genera ted by a r ep r e sen t a t i on which keeps t rack of the physical p roper t i e s of the mate r i a l s . Th i s work is useful no t only to a r t i s t s who w a n t to p a i n t in t rac- t ively, bu t also for a u t o m a t e d render ing of n a t u r a l (or non- realist ic) scenes. As t echn iques like r ay - t r ac ing ex t end the abi l i ty of c o m p u t e r s to render scenes wi th p h o t o g r a p h i c ex- ac t i t ide , the re will be a c o m p l e m e n t a r y a d v a n c e m e n t of tech- n iques which allow c o m p u t e r s to su99est scenes wi th ar t i s t ic

abs t r ac t i on .

Permission to copy without fee al[ or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and / o r specific permission.

© 1986 A C M 0 - 8 9 7 9 1 - 1 9 6 - 2 / 8 6 / 0 0 8 [ 0 2 2 5 $ 0 0 . 7 5

1 . 1 P r e v i o u s w o r k

In W h i t t e d [11], an u n c h a n g i n g an t i -a l i ased image is d ragged to d raw a s m o o t h glossy tube . P a i n t s y s t e m s us ing i n p u t de- vices wi th th ree or more degrees of f r eedom (say, pos i t ion and

pressure) allow the user to vary some p a r a m e t e r of the b rush p a t t e r n (say, r ad ius or hue of a solid circle) as they pain t . Lewis 151 descr ibes s t o c h a s t i c and f r e q u e n c y - d o m a i n repre- s e n t a t i o n s of t ex tu re , b u t t hese t echn iques do not adequa te ly render t he effects at t he bounda r i e s of d iscre te s t rokes .

Greene [4] descr ibes an inpu t device called the "d rawing p r i sm" which digi t izes the image of a real b ru sh (or o ther objec t ) m a k i n g opt ica l con t ac t with a t r a n s p a r e n t p r i sm. Al- t h o u g h t he resu l t ing images are real is t ic , t he s y s t e m h as no r ep re sen t a t i ona l a b s t r a c t i o n h igher t h a n t he pixel level. Th e s y s t e m descr ibed in th is thes i s s i m u l a t e s a b r u s h s t roke us- ing a h ie ra rchy of r ep r e sen t a t i on , a l lowing repea tab i l i t y and

e x p e r i m e n t a t i o n a t m a n y levels o f control .

1 . 2 S n m i - e P a i n t i n g

T h i s research was inspi red by the t r ad i t i ona l J a p a n e s e a r t known as sumi-e. A l t h o u g h the re are a wide var ie ty o f snmi-e pa in t i ng s ty les , one seems a pa r t i cu la r ly good c a n d i d a t e for c o m p u t e r s imu la t i on . P a i n t i n g s in t he bokkotsu s ty le are

charac te r i zed by a few well-placed s t rokes on a l ight back- g round . P ic tu res wi th h u n d r e d s or more s t rokes m a y b eco m e prac t ica l some day, bu t for now bokkotsu sumi-e is appea l -

ing as a model because evoca t ive p ic tu res m a y be m a d e in black and whi te , and wi th only a few strokes. T h e bokkotsu s ty le emphas i ze s the qua l i ty of each s troke; th is focuses the a t t e n t i o n on the processes and ma te r i a l s involved in the con- s t ruc t i on of each s troke.

F igure 1: An e x a m p l e of Sumi-e: ' eShrimp and Leaf"

225

S I G G R A P H '86

An e x a m p l e of c o m p u t e r - g e n e r a t e d s u m i - e 1 can be seen in F igure 1. The p ic tu re is one f r a m e f rom an a n i m a t e d se- quence (descr ibed below in Sect ion 6). It c o n t a i n s 17 s t rokes , each defined as a spl ine wi th be tween 3 and 8 cont ro l po in t s . It is an t i -a l iased , and was gene ra ted on an 8-bit 640 × 480 pixel f r ame buffer. T h e des ign was d r a w n f ree -hand by t he a u t h o r , in te rac t ive ly , us ing a mouse .

2 T h e R e p r e s e n t a t i o n

T h e key to a successful i m p l e m e n t a t i o n is choos ing the r igh t r e p r e s e n t a t i o n . In a t t e m p t i n g to s i m u l a t e b rushes , t he re is a broad s p e c t r u m of poss ible r ep r e sen t a t i ons , r ang i ng f rom the s imp le to the complex , which would have a co r r e spond ing degree of rea l i sm and c o m p u t a t ional expense . In th i s sec t ion , ] d i scuss t he basic r ep r e sen t a t i ona l un i t s and m y r easons for choos ing t h e m .

2 . 1 T h e O b j e c t s

All o f t h e code is w r i t t e n in Ze ta l i sp us ing f l a v o r s , an object - o r i en t ed p r o g r a m m i n g style . T h e bas ic r e p r e s e n t a t i o n a l u n i t s a re the re fo re f lavors, or c lasses of objects . T h i s c rea tes a use- ful m o d u l a r a b s t r a c t i o n which al lows the user to deal wi th the m a n y p a r a m e t e r s of d r a w i n g in a m a n a g a b l e and s t r u c t u r e d h ie ra rchy .

T h e y are:

1. B r u s h - a c o m p o u n d object c o m p o s e d of br i s t les 2. S t roke - a t r a j ec to ry of pos i t ion ~nd pressure 3. Dip - a desc r ip t ion of the ini t ial s t a t e of a c lass of

b r u s h e s 4. P a p e r - a m a p p i n g on to the d i sp lay device

Br is t les , which do a lot of the work of the b r u s h , are also ob jec t s in the i r own r ight , bu t the i r desc r ip t ion and defini t ion is i n t i m a t e l y connec t ed wi th t h a t of the b ru sh .

2 . 1 . 1 T h e B r u s h

A b r u s h can be t h o u g h t of as a one -d imens iona l a r r ay of bris- t les , each of which ha s i ts own ink supp ly and pos i t ion re la t ive to t h e b r u s h handle . T h u s , t he b r u s h h a s a one -d imens iona l " foo tp r in t " (like a windsh ie ld wiper) , and is moved so t h a t it is a lways pe rpend i cu l a r to t he p a t h of the s t roke .

As the b ru sh is m o v e d t h r o u g h t he t r a j ec to ry specified by t h e s t roke , two per iodic c o m p u t a t i o n s are per formed:

• T h e s t a t e of each br i s t le is u p d a t e d .

U p d a t i n g t he br i s t les cons i s t s of eva l ua t i ng one or more code f r a g m e n t s ( " ru le s" ) which modi fy the color, ink q u a n t i t y , re la t ive pos i t ion , or any o the r p rope r ty for each brist le .

• An image c o m p u t e d f rom the br i s t les is t r ans fe r red to the paper .

A single br is t le c o n t r i b u t e s to t he image only if two cond i t ions bo th hold: it is appl ied to t he pape r wi th

IPronounced ~ov-tne-ay, it comes from the Japanese words "sumi," the black ink used in calligraphy, and ~ef meaning picture. Sato's work 18] has many examples and discusses the history, symbolism, and techniques of traditional and modern sumi-e. It includes the famous Muslard Seed Garden Manual of Painting, n compendium of 1000 years of sum*-e experience and technique first published in China in 1679.

sufficient pressure , and it h a s ink r ema in ing . A drop le t of ink co r r e spond ing to t h a t b r i s t l e ' s color a t t h a t po in t in t ime is added to t he pape r by send ing a m e s s a g e to a pape r object (see Sect ion 2.1.4).

Each br is t le ha s a color: for s u m i t he color is s imp ly a shade of gray, r ep resen ted as a f rac t ion be tween 0 and 1. It is a s s u m e d t h a t all t he ink on a given br is t le is of t he s a m e color; however , ne ighbor ing br i s t les m a y be of different colors.

2 . 1 . 2 T h e S t r o k e

A s t roke is a se t of p a r a m e t e r s (e.g. pos i t ion and pressure) wh ich evolve as a func t ion of an i n d e p e n d e n t var iable . Th i s m a y be t h o u g h t of as e lapsed t ime , oi" t he d i s t an ce a long the s t roke; any m o n o t o n i c a l l y increas ing var iab le will do. ] call th i s var iable " t i m e " , and r ep re sen t it wi th t he s y m b o l S (since T al.ready h a s a special m e a n i n g in LISP) . I ts va lue is an a p p r o x i m a t i o n of t he d i s t a n c e a long t he s t roke .

Since the re is no specia l i n p u t h a r d w a r e (o the r t h a n a keyboa rd and a mouse ) cu r r en t l y a t t a c h e d to our Lisp m a- ch ines , t he s h a p e of t h e s t roke is d e t e r m i n e d by a spl ine of 2D c o o r d i n a t e s specifed us ing the mouse , c l icking once to spec- ify each cont ro l poin t . For each cont ro l po in t , t h e user can specify t he p res su re m a n u a l l y w i th t he keyboa rd . T h e spl ine i t se l f is a connec t ed ser ies of line s e g m e n t s suff ic ient ly smal l to give the i l lusion of a s m o o t h curve.

Pos i t ion and p re s su re s a m p l e s and the spl ines der ived f rom t h e m are s to red in the "s t roke" object . T h e user can edi t an incorrec t s t roke , or select a different b rush or dip for t he s a m e t ra jec tory .

2 . 1 . 3 T h e D i p

In t r ad i t i ona l Or i en t a l p a i n t i n g and ca l l igraphy, a complex t e x t u r e of color and u n e v e n d i s t r ibu t ion of ink can be ap- plied to t h e b rush . T h i s can se t up the p a t t e r n s of l ight and d a r k n e s s wh ich can m a k e a s imple s t r a i g h t s t roke look like a cyl indr ical s e g m e n t of s m o o t h b a m b o o , or m a k e a cliff r is ing ou t of t he ocean seem to be covered wi th moss on top. By s e p a r a t i n g the a b s t r a c t i o n of dip f rom b rush , one can use the s a m e b ru sh for a wide var ie ty of s t rokes and effects, j u s t as in real s u m i . If one selects a pa r t i cu la r b ru sh a n d stroke~ one can e x p e r i m e n t wi th different d ips to ach ieve exac t ly the desired effect.

Since m o v i n g a b ru sh t h r o u g h a s t roke uses up t h e ink and can change t he pos i t ion and color of the br is t les , t h e dip m u s t car ry e n o u g h i n f o r m a t i o n to res tore t h e b r u s h to i ts ini- t ial s t a t e (or a suff icient ly s imi la r s t a t e ) , so t h a t s t rokes can be repea ted . T h i s can be a n y t h i n g f rom us ing a s imple rule to s to r ing an expl ic t s n a p s h o t of t he s t a t e of each brist le . Such a rule is a p rocedu re which ha s access to p a r a m e t e r s such as the pos i t ion of each br is t le w i th in the b ru sh , or user- specified p a r a m e t e r s like b lo tch iness or s m o o t h n e s s . Dipp ing a b r u s h execu te s t he p rocedure a n d / o r copies t he s to red bris- t le p a r a m e t e r s . R a n d o m n e s s can be in t roduced a t t h e t im e of c r ea t ing the dip, a n d / o r a t each act of d ipping .

2 . 1 . 4 T h e P a p e r

T h e pape r object is respons ib le for r ender ing t he ink as it comes off the b rush . As each br is t le decides to i m p r i n t itself, i t s ends a m e s s a g e to the paper ind ica t ing i ts pos i t ion and o the r re levan t p a r a m e t e r s . T h e paper t hen reacts , usua l ly by

226

Dallas, August 18-22 Volume 20, Number 4, 1986

render ing a single dot of app rop r i a t e color at the app rop r i a t e poin t .

T h e pape r concept is useful because it p r e s e n t s an ab- s t r a c t i on wh ich al lows t he s y s t e m to run on f r ame buffers of var ious reso lu t ions and dep ths . An a rb i t r a ry t ex tu re can be m a p p e d over the s t roke in several w a y s to s i m u l a t e tex- t u r ed pape r s (see Section 4.5), us ing an a l g o r i t h m s imi la r to conven t iona l t e x t u r e - m a p p i n g . T he paper a b s t r a c t i o n also has the po ten t i a l of model l ing such effects as the we tne s s or abso rp t ive p roper t i e s of real paper , b u t I have not ye t imple- m e n t e d such behavior , l d i scuss s o m e of the poss ibi l i t ies in Section 7.5.

Cur ren t ly . the user can d raw on f r ame buffers wi th e i ther 1, 8, or 24 b i t s per pixel, at e i ther N T S C (640 ~ 480) or h igh (1280 × 1024) resolu t ion . P a p e r s of a rb i t ra r i ly h igher resolu t ion can be s i m u l a t e d because of t he s u p e r - s a m p l i n g pa t ch provided for an t i -a l i as ing (see Sect ion 3.2).

2 . 2 W h y t h i s r e p r e s e n t a t i o n ?

A l t h o u g h 1 have identif ied four ma j o r a b s t r a c t i o n s and a hos t of effects (descr ibed in Sect ion 4) wh ich can be c rea ted w i th t h e m , it is still t oo ear ly to pin down the i r exac t specifica- t ions . R a t h e r t h a n des ign a s y s t e m which exac t ly e m u l a t e s , say, a camel ha i r b r u s h d ipped in a pa r t i cu la r b r and of ind ia ink, ] chose to des ign a f r amework in which m a n y ca tegor ies of pa in t - l ike m e d i a can be expressed.

Anyone who h a s ever walked in to an a r t s tore can a t t e s t to the fact t h a t - to t he novice - the re s e e m s to be a bewi lder ing n u m b e r of degrees of f r eedom to con t ro l in a r t i s t ic media . It is i m p o r t a n t to choose a r ep r e sen t a t i on t h a t is m o d u l a r for the following reasons :

Q T h e user can become fami l ia r wi th a sma l l reper to i re of famil iar tools. For examp l e , different b ru shes can be used and re-used over t he s ame s t roke to explore var ious effects. A cer ta in d ip or paper , once per fec ted , can be saved for la ter use.

• Since the s i m u l a t i o n is based on a m o d u l a r and hierar - chically organized set of effects, a spec t s o f the s imu la - t ion can be replaced or a u g m e n t e d wi th m o r e sophis t i - ca ted a lgo r i thmic mode l s as they are developed.

• T h e s a m e p ic ture can be rendered a t m a n y levels of complex i ty , f rom quick d ra f t s to final images , by se- lectively " t u r n i n g on" different effects i n d e p e n d e n t l y of each o ther .

3 The I m p l e m e n t a t i o n

Thi s sec t ion descr ibes t h e c o m p u t a t i o n of t he b r u s h ' s mo- t ion. Ear l ier work involved b ru shes wi th t wo-d i mens iona l foo tp r in t s , bu t l found I could save a g rea t deal of c o m p u - t a t i on w i t h o u t m u c h loss of genera l i ty us ing a 1D foo tpr in t . A d i scuss ion of t he issues leading to th i s decision, p lus t h e de ta i l s of t he i m p l e m e n t a t i o n wi th the 2D foo tp r in t is in [10].

T h e s t roke is defined by the user as a l ist of pos i t ion and p re s su re s amples . B o t h pos i t ion and pressure are in te rpo- la ted us ing a cubic spl ine to yield a series of nodes (F igure 2). T h e w id th of t he s t roke is c o m p u t e d for each node by a func t ion of t he p res su re at t h a t node, and t h e region be tween two nodes is covered by a quadr i l a t e ra l defined below.

F igure 2: A s t roke defined by 4 control po in t s wi th in te rven- ing nodes gene ra ted by a cubic spline. T h e a rea covered by the s t roke is a p p r o x i m a t e d by quadr i l a t e ra l s .

Once the quad r i l a t e r a l is found , a general ized po lygon in t e rpo la t ion a l g o r i t h m is used to ass ign each pixel w i th in t h a t po lygon

* i ts pos i t ion in " t ime" a long the s t roke , and

• the br is t le wh ich passes nea re s t to it.

T h e fo rmer is used to sor t t he pixels in the order t h a t t h e b ru sh sweeps over t h e m , and the l a t t e r is used to c o m p u t e the color for t h a t pixel.

It is i m p o r t a n t to no te t h a t conven t iona l polygon-f i l l ing a l g o r i t h m s are no t useful to me here. Th i s is because l need to d r aw each pixel in chronological order in order to c a p t u r e the evo lu t ion of the br i s t les as t hey move a long the s t roke.

3 . 1 D e t a i l s

T h e user first specifies the s t roke pa th . Th i s is r ep resen ted as t he N nodes (X ,Y ,P ,S ) i for i = 0 . . . . . ( N - 1). T h e values A' ,Y, and P represent position and pressure . S is an ap- p r o x i m a t i o n of t he d i s t ance t rave led a long the curve , where So = 0. The b r u s h ' s cen te r m o v e s a long t h e line s e g m e n t s connec t ing consecu t ive nodes (X, Y)i , as c o m p u t e d by Bre- s e n h a m ' s a l g o r i t h m [7] for d r awing line s e g m e n t s .

On a s y s t e m wi th no c o n t i n u o u s p ressu re - sens i t ive i n p u t device, t he p a t h m a y be derived f rom a cubic spl ine wi th n cont ro l po in t s . Each po in t is a t r iple (x,y,p)i speci fy ing loca t ion and p ressu re a t t he jeh poin t . A f o u r t h value ap- p r o x i m a t i n g t he d i s t ance a long t he curve, sj, is c o m p u t e d for t h a t poin t .

J ~o = o, sj = ~ ~ ( ~ k - x k - , ) 2 + ( ~ - ~_,)~

k=]

From th is , two 2D cubic spl ines are created. One comes f rom (x, y)j and gene ra te s (X, Y) i , t h e o the r f rom (p, s ) i and gene ra t e s (P, S)i.

Let us call two such nodes A and B. Be tween t h e m , a s e g m e n t (A-B) is created. If AB is no t the las t s e g m e n t , t h e nex t po in t is called C .

F

E C

H

F igure 3: C o n s t r u c t i o n of t he po lygon connec t ing nodes A and B.

2 2 7

~. S I G G R A P H '86

For each segment, a quadrilateral ( E F G H) is constructed which has the following properties [Figure 3]:

• A bisects E H • B bisects FG

b i

tEH] is the width computed from the pressure at A I F

!ffG-I is the width computed from the pressure at B 4 F

• FG bisects / A B C .

One exceptional case is when the lines E H and FG actu- ally intersect /Figure 4]. This can happen if A B is relatively short compared to the width EH. Since the polygon inter- polation algorithm I use insists on being handed vertices in clockwise order, one cannot simply pass on the quadrilateral E F G H . 1 call this the "bow-tie" case, and l handle it by par- titioning the bow tie into two triangles, and rendering each one independently. Note that one cannot simply swap the of- fending vertices, since the chronological order of the vertices must be preserved.

F

A H

~P

Figure 4: The annoying bowtie case.

Once the polygon's vertices are found, three properties are generated for each pixel using 2D interpolation algorithm.

1. Its position on the frame buffer (X ,Y) . This is gener- ated in the course of the interpolation.

2. Its position along the stroke (S). This is done by inter- polating (SAD SB, SB, SA) on polygon E F G H .

3. Its position across the brush (B). This is done by in- terpolating (1, l , 0, 0) on polygon E FG H.

As the pixels are generated, they are sorted chronolog- ically (by S) into a temporary array of length (SB -- SA). Then, as the brush moves and updates, all pixels belonging to that portion of the stroke can be drawn. For each pixel, its abstract brush position B (where 0 < B < 1) is used to determine the nearest responsible bristle(s).

3 . 2 A n t i - a l i a s i n g

Although the stroke is broken into polygons, one cannot merely anti-alias the polygon edges, since the brush could theoret- ically change anywhere, at any time; i.e. every pixel could be an edge. So, anti-aliasing is done by supersampling. The brush draws on a patch of virtual paper at a higher reso- lution than the frame buffer. For each polygon, a 2D ar- ray the size of the polygon's bounding-box is allocated, and the corresponding image of the paper is copied to the patch. This idea is similar to the high-resolution patch used in [ll], except that the brush's image cannot be pre-computed like Whit ted 's , since it is constantly changing. The polygon is rendered at high resolution onto the patch, which is then sampled and copied back to the frame buffer.

3 .3 E f f i c i e n c y

The computational time consumed by the algorithm can be separated into two parts:

• The serial part; this is the computation of the stroke geometry, e.g. computation of the polygon vertices and edges.

• The parallel part; this can be broken into two parts:

- Each bristle executes the evolution rules to deter- mine its next state.

- Each pixel consults the brush to determine what color it should become.

From running several informal timing benchmarks, it seems that about 90% to 99% of the computation on a serial ma- chine is occupied by the parallel part of the algorithm, except for pathologically small strokes and brushes. Thus, although the polygon-vertex computation seems complex, it occupies an insignificant amount of time compared to the rendering.

Of the parallel part, the ratio of time between the two parts is very much a function of how big the brush is, as measured both in bristles and in pixels. Another important factor is the complexity of the evolution rules.

4 E f f e c t s

In this section, 1 will describe some of the effects one can control by changing different parameters of the simulation. Although there may seem to be a bewildering myriad of pa- rameters to control, i t 's important to recognize that each parameter has an intuitively recognizable function, and its effect on the image can he appreciated with a minimum of experimentation.

4 .1 I n k Q u a n t i t y

The ink supply on each bristle is assumed to be a reservoir of a finite quantity of fluid, which gets replenished each time the brush is dipped. The quantity is decreased as the brush moves through the stroke, and eventually the bristle runs out. When the quanti ty drops to zero, that bristle no longer contributes to the image on the paper.

If a scratchy breakup at the tail of each stroke is desired, the dip should put just the right amount of ink on the brush, including selecting a few bristles to be short-changed so they run out early [Figure 5]. If the stroke is known at the time of the act of dipping, its length is used to help determine the quanti ty of ink deposited on the bristles. There are parame- ters which control how many bristles get short-changed, and by how much, either as a fraction of the total stroke length or in units of absolute distance.

Figure 5: Different quantities: (A) 50% of the bristles are approx. 33% dry. (B) 75% of the bristles are approx. 50% dry.

228

Dallas, August 18-22 Volume 20, Number 4, 1986

4 . 2 I n k C o l o r

Each br is t le has a color: for sumi t h e color is s imply a shade of gray, r ep resen ted as a f rac t ion be tween 0 and 1. It is a s s u m e d t h a t all t he ink on a given bris t le is of t he s a m e color; however , ne ighbor ing br is t les m a y be of different colors.

F igu re 6: Different colors: (A) C o n s t a n t (B) Linear (C) User-specif ied

The d i s t r i bu t ion of color across the b r u s h m a y be specif ied a s c o n s t a n t , or a l inear r a m p f rom one value to a n o t h e r , or as a n explici t l ist of a rb i t r a ry va lues [Figure 6 I. A l t h o u g h th i s d i s t r i bu t ion m u s t be specified for the beg inn ing of the s t roke , t he re are severa l w a y s of t h i n k i n g a b o u t how t h e color evolves over the s troke:

• A d i s t r i bu t ion is specif ied for bo th t he s t a r t and end of the s t roke. A t any po in t in the middle of the s t roke , the color of a given br is t le is l inearly i n t e rpo l a t ed be tween the s t a r t i n g and end ing values specified for it IFigure 7 A, B). Th i s idea m a y be ex t ended to general ized distr i- bu t ion samples a t a rb i t r a ry po i n t s in {he s troke.

• F rom the s t a r t i n g d i s t r i bu t ion , diffusion m a y be s im- u l a t ed by s m o o t h i n g the colors of ne ighbor ing br is t les [Figure 7 C,D]. Each br is t le u p d a t e s i ts color accord ing to a par t ia l i n t e rpo la t ion . For examp l e , if

- Ci, is t he color on t he i t h br is t le at t i m e t,

- D is a speed-of-di f fus ion p a r a m e t e r be t ween 0 and 1 (1 is rap id diffusion),

- and t he br is t les are a s s u m e d to be regular ly spaced,

ThenC, ,+~ = C , , ( 1 - D ) + ( c ' - "2+c '+ ' , )D .

• A general ized evo lu t ion a l g o r i t h m can be suppl ied [Fig-

ure 7 E, F]. T he color on a br is t le m a y be a func t ion of b r u s h pressure , d i s t ance f rom the origin, or even the q u a n t i t y of the ink r e m a i n i n g (see Section 4.3).

F igure 7: Color effects: (A) S t a r t / e n d in t e rpo la t ion f rom one r a m p to a n o t h e r (B) In te rpo la t ion f rom spike to no tch (C) Fas t color diffusion (D = .5) (D) Slow diffusion (D = .1) (E) R a n d o m ( B r o w n i a n ) e v o l u t i o n of color (F) "Ink s tea l ing" evolu t ion of quan t i t y .

In add i t ion to or i n s t ead of t he above p h e n o m e n a , a pa t - t e rn can be t e x t u r e - m a p p e d on to the s t roke (see Sect ion 4.5).

Once t he color of t he ink on t he br is t le is decided, t h e color to place on the pape r must . be c o m p u t e d . T h e p ap e r m a y a l ready be colored due to e i ther the pape r ' s n a t u r a l tex- tu re or previous ly depos i t ed ink. The user m a y supp ly a color combination function of two or t h r ee i n p u t s to be e v a l u a t e d

each t ime a given br is t le a t t e m p t s to d raw on a pa r t i cu l a r pixel of the paper ; t he i n p u t s are the ink color (Ci), t he color of t he pape r a t the po in t to be d r a w n upon (Cp), and an op- t iona l va lue der ived f rom the t e x t u r e - m a p p i n g ar ray , if the re is one.

For sumi t h e defau l t func t ion used is a very s imple one: t he d a r k n e s s a t t he in te r sec t ion of severa l s t rokes is a s s u m e d to equal the d a r k n e s s of t h e da rke s t s t roke (e.g. Cpt+j = max (C~,, Cp,))

4 . 3 E v o l u t i o n o f Q u a n t i t y a n d C o l o r

Jos t l ing of ne ighbor ing br is t les s o m e t i m e s t r ans fe r s ink a m o n g t hem; th i s affects bo th t he q u a n t i t y and color of the bris- t les concerned. Th i s is mode led by t h ink ing of t h e b ru sh as a cel lular a u t o m a t o n [12] wi th a smal l p rocedure (rule) for q u a n t i t y and color t ransfe r . As the b rush m o v e s across the page a bi t a t a t ime , all br is t les repea ted ly execute the s a m e rule, which can refer to t he p a r a m e t e r s of each br is t le and i ts i m m e d i a t e ne ighbors . It m a y c o m p u t e a new value for any of t h a t b r i s t l e ' s p a r a m e t e r s and modi fy it accordingly.

For examp le , one rule can allow a near -d ry bris t le to run ou t of ink, t hen t empora r i l y "discover" a new supply (ei- ther by s tea l ing f rom a ne ighbor , or j u s t con ju r ing it o u t of nowhere) to crea te i s lands of ink and wh i t e spac e in the mid- dle of t he s t roke [Figure 7 F]. Inco rpora t ing a,n e l emen t of r a n d o m n e s s in to the rules can give rise to r ich t ex tu res . On the o the r hand , avo id ing r a n d o m n e s s m a y be necessary in ap- p l ica t ions like s o m e k inds of a n i m a t i o n , where t h e user w a n t s the complex t e x t u r e of s t roke to be cons i s t en t f rom f rame to f r ame (see Section 6).

4 . 4 P r e s s u r e

T h e p ressure on a pa r t i cu la r br is t le is a func t ion of the ge- o m e t r y of the b rush and the overall p ressure on the b rush a t a cer ta in poin t in t he s t roke.

C h a n g i n g the applied p res su re du r ing t he s t roke can have two different k inds of effects:

* Spreading. Pre s s ing ha rde r can sp read the br is t les fur- ther apa r t .

• Contact. Pres s ing ha rde r can b r ing more br is t les in to con t ac t wi th the paper .

Unde r sp read ing , each br i s t le ' s d i s t ance f rom th e b r u s h cen te r is an a rb i t r a ry func t ion of t he appl ied pressure . By de- fau l t , d i s t ance is l inearly p ropor t iona l to p ressure , bu t so m e in t e re s t ing effects can be d e m o n s t r a t e d by explor ing o the r r e l a t ionsh ips [Figure 8].

One can also cons ider t h a t g rea te r overal l p r e s su re b r ings m o r e br is t les in to c o n t a c t w i th the paper [Figure 9]. A value is a ss igned to each br is t le wh ich r ep resen t s t h e m i n i m u m b rush p ressure necessary to b r ing it in to con t ac t w i th the paper . For e x a m p l e , to s i m u l a t e a round b ru sh of r ad iu s l , each br is t le gets a p r e s su re - t h r e sho ld p ropor t iona l to the a rcs ine of i ts d i s t ance f rom the cen te r of the b rush .

2 2 9

~,~ S I G G R A P H '86

Figure 8: Spreading under pressure: (A) Constant width (B) Width o¢ pressure (C) Width quantized by user-supplied function

Figure 9: Two interpretations of pressure: (A) More pressure spreads bristles (B) More pressure brings more bristles into contact (C) A combination of these two effects.

Intermit tent contact with the paper near the pressure threshold is simulated by adding a rule which causes per- turbations (either overall, or for individual bristles) in two paramete.

• Changing the brush's pressure implies one's hand is os- cillating.

• Changing the the pressure-threshold of a bristle implies the geometry of the brush is changing.

Although these parameters have different meanings, the image ultimately depends on the difference between them; so it doesn' t mat te r which modified as long as one is con- sistent. A more realistic test for determining contact might take into account the orientation of the brush and hysteresis (stickiness).

4 .5 T e x t u r e M a p p i n g

Some interesting effects can be realized by mapping a texture onto the image of the stroke [Figure 10]. There are at least two ways of computing the mapping:

Figure 10: Texture mapping: (A) Textured paper (B) Tex- tured by smiley-face paper (C) Texture mapping with spread- ing bristles (D) Texture mapping with pressure-threshold bristles.

A rectangular array representing the texture of the pa- per is mapped by a straightforward fiat tiling. When a bristle a t tempts to draw ink of a certain color on a given pixel, the array element corresponding to that pixel is used.

A one or two dimensional array is mapped along the long axis of the stroke {this is only used in the im- plementations where the brush has a one-dimensional footprint). The array element used corresponds to how far along the stroke the brush has travelled. For ex- ample, a simple 1D texture (say, a sine wave) mapped onto a curvy stroke gives the impression of banding similar to a raccoon's tail. If the texture map is two- dimensional, the bristle 's radial distance from the brush center is used to compute the array index in the second dimension.

Once a value is supplied by the texture array, it is used in the user-supplied color combination function (see Section 4.2). For sumi the texture value is a number (usually a fraction between 0 and 1) which is multiplied by the ink color to selectively at tenuate it before applying it to the paper.

5 T h e U s e r I n t e r f a c e

The prototype system is not intended to be a general-purpose paint program. As such, it lacks most of the useful function- ality and well-designed user interface users have come to ex- pect from paint programs. Instead, effort was concentrated on exploring new rendering algorithms and effects.

5 .1 S y s t e m s p e c i f i c s

The system runs on a Symbolics 3600 Lisp Machine with a 1280 x 1024 × 24 bit frame buffer, configured for 8-bit per pixel grayscale. All of the code is written in Zetalisp using flavors, using an object-oriented programming style. The rules which govern the evolution of the brush are pieces of Lisp code ("methods" in Zetalisp) associated with a particular flavor of brush. These rules are executed as the brush moves along its path, and the code in them can freely refer to and modify any parameters.

5 .2 D r a w i n g

The prototype system takes up to a minute or two to ren- der a stroke, depending on brush complexity and the super- sampling ratio for anti-aliased strokes. Although this is too slow for real-time interactive drawing, the user enters and edits the strokes' paths interactively. The input consists of discrete samples of position and pressure, which are then smoothed using a cubic spline by the rendering algorithm. The user uses a mouse to position the points of the stroke with one hand, while the other hand specifies the pressure with the keyboard.

The user can dynamically select from among 16 different levels of pressure by pressing combinations of four keys on the standard keyboard. The cursor responds to this in real

time by growing or shrinking accordingly, depending on the pressure-to-width algorithm selected by the user.

230

Dallas, August 18-22 Volume 20, Number 4, 1986

6 A n i m a t i o n

One of the most important motivations for this work is the hope that creating reproducible brush strokes will allow paint- ings to be animated.

The code supporting animation right now is rather crude. It is basically a two-dimensional keyframing system which al- lows the user to specify key shapes of brush strokes as they change over time. The position and pressure of each control point of the stroke is interpolated between the key frames, using a spline for non-periodic motion and a generalized si- nusoid for periodic motion. The same brush and dip is used for any given animated stroke.

The shrimp shown in Figure 1 was animated from four key frames, spline interpolated over the total 92 frame se- quence. Each frame took about one minute to render. With the antennae and legs waving around, the tail kicking, and the ripples flowing away from the leaf, the resulting anima- tion is very lifelike.

7 F u r t h e r w o r k :

7 .1 B e t t e r i n p u t m e t h o d s

Without an input device as expressive as a real brush, the current environment isn't very user friendly. There are many kinds of input devices offering three or more degrees of free- dom which might be adapted for manually entering strokes. At the MIT Media Lab, we are exploring force sensitive touch- screens [6], LED-based body trackers I3], and magnetic point- ing devices [9]. Other possible input devices include touch- sensitive tablets [2! and the drawing prism 14].

7 .2 B e t t e r r e n d e r i n g h a r d w a r e

With the advent of parallel computers, the drawing of the most sophisticated of strokes should be possible in real time. This is because almost all of the computation in the algo- r i thms described here are local, that is, dependent only on an immediate neighborhood of bristles or pixels, and thus is well-suited to implementation on machines using parallel architectures.

7 .3 E x p l o r i n g r u l e s

More experimentation is needed to build a good-sized library of rules. Hopefully, subjective properties like "blotchiness", "dryness", or "clumpiness" can be controlled by adding a rule and setting a parameter or two. New kinds of rules xvill result in innovative brushes, as well as realistic models of traditional watercolor brushes.

7 .4 R e a l c o l o r

For simplicity, I stayed with monochrome ink even though the frame buffer I used has full 24 bit color. A useful extension would be to allow the user to experiment with a virtual brush laden with various colors. A more complicated rule would describe the behavior of paint mixing. Real electronic paint can change color as a function of thickness of application or chemical reaction with the brush, paper, or other strokes.

7 .5 P a p e r e f f e c t s

The wetness and absorptive properties of the ink or paper can be described by specifying the area of the paper covered by each bristle, and an ink redistribution function associated with the paper. The former corresponds to the pre-filtering and the latter to post-filtering steps in anti-aliasing. In ad- dition to the usual blurring (low-pass filter) operations, one could use a simple asymmetrical fractal to simulate the forked bleeding that capillary action sometimes causes on dry pa- pers.

7 .6 S p l a t t e r

A bit of splatter from a heavily-laden brush with stiff bristles pulled briskly around a corner might be represented as a rule which gets activated when the brush velocity or accelleration surpasses a certain threshold. It then places a fractal distri- bution of splattered, fuzzy dots on the paper as a function of the ink supply, trajectory, and pressure on the brush.

7 .7 M u s i c a n d p a i n t i n g

An appealing analogy to the stroke is the contour of a musical note over time. Each stroke is a set of time-varying parame- ters (like position and pressure, or loudness and timbre). A cluster of strokes can evoke a recognizable image, much as a collection of notes create a chord or arpeggio.

Occasionally, when I am asked about the limits of real- ism in my simulation, ] am reminded of similar questions asked of builders of electronic instruments. The answer, of course, is that there is room in electronic media for both ac- curate reproduction of physical phenomena, and for creative exploration with totally new forms of expression which take advantage of the differences inherent in the new media.

7 .8 3 D s t rokes

Perhaps the strokes themselves can be liberated from the 2D quality of paper, and a technology of 3D paintbrushes can be realized. Non-computer techniques come to mind, includ- ing sweeping a lit taper through a room to leave a trail of smoke, or "drawing" in an aquarium filled with a viscous gel using a long hypodermic filled with ink. All the issues of describing the evolution of texture through the stroke re- main. With stereoscopic displays [9] or computer-generated holograms [1], one will be able to create tenuous sculptures far more delicate than currently possible. One could even imagine folding translucent paper into origami shapes which define plane fragments on which these brush strokes lie.

8 C o n c l u s i o n

To get realistic brush strokes, one must simulate the phenom- ena which gives rise to them. The tough part is modeling the behavior of individual bristles, which we do with rules which execute each time the brush moves. To produce an anti- atiased image, the brush's image is drawn onto a small patch which is sampled and incrementally copied to the page.

When an animal, plant, or river can be represented by a few deft strokes, perhaps under some circumstances a brush representation can replace a polyhedral one. Whereas poly- hedra are good representations of analytic objects, and polyg- onal fractals are good for largely amorphous ones, there is a

231

~. S I G G R A P H '86

middle classification of things too rich for one and too struc- tured for the other. Given the computational complexity and storage expense of representing a warm, fuzzy bunny as a skeleton of faceted polyhedra covered with skin polyhedra and particle generated hair, perhaps representing it instead as a collection of brush strokes would result in faster rendering, more compact storage, and a more aesthetically appealing image.

9 Acknowledgements

This work was supported in part by a grant from Apple Com- puter, Inc., and an equipment loan from Symbolics, Inc. Karl Sims suggested the use of and provided the polygon interpo- lation routine.

R e f e r e n c e s

[1] Benton, Stephen A., "Holographic Displays - - A Re- view," Optical Engineering 14, 5, Sept-Oct 1975.

[2] Buxton, W., Hill, R., and Rowley, P., "Issues and Techniques in Touch-Sensitive Tablet Input," Com- puter Graphic~ 19~ 3, July 1985, pp. 215-224.

[3] Ginsberg, C., and Maxwell, D., "Graphical Mari- onette," Proc. A CM SIGGRAPH/SIGART Workshop on Motion, April 1983, pp. 172-179.

[4] Greene, Richard, "The Drawing Prism: A Versatile Graphic Input Device," Computer Graphics 19, 3, July 1985, pp. 103-110.

[5] Lewis, John-Peter, "Texture Synthesis for Digital Painting," Computer Graphics 18~ 3, July 1984, pp. 245-251.

[6] Minsky, Margaret, "Manipulating Simulated Objects with Real-world Gestures using a Force and Position Sensitive Screen," Computer Graphics 18~ 3, July 1984, pp. 195-203.

[7] Rogers, David F., "Procedural Elements for Computer Graphics," McGraw-Hill, New York, 1985.

[8] Sato, Shozo, "The Art of Sumi-e" (in English), Kodan- sha International, Tokyo, 1984.

[9] Schmaudt, Chris, "Spatial Input/Display Correspon- dence in a Stereoscopic Computer Graphic Worksta- tion," Computer Graphics 17~ 3, July 1983, pp. 253- 257.

I10] Strassmann, Steve, "Hairy Brushes in Computer- Generated Images," M.S. Thesis, MIT Media Labora- tory, June 1986.

[111 Whitted, Turner, "Anti-aliased Line Drawing Using Brush Extrusion," Computer Graphics 17~ 3, July 1983~ pp. 151-156.

[12] Wolfram, Stephen, "Cellular Automata as Models of Complexity," Nature 311~ 4, 1984 , pp. 419-424.

232