Revisiting Visual Interface Programming:Creating GUI Tools for Designers and Programmers

Stéphane Chatty1 Stéphane Sire1 Jean-Luc Vinot2Patrick Lecoanet2 Alexandre Lemort1 Christophe Mertz1,2

1IntuiLab 2CENAPrologue 1, La Pyrénéenne 7 avenue Edouard Belin

31372 Labège Cedex, France 31055 Toulouse Cedex, France{chatty,sire,lemort,mertz}@intuilab.com {lecoanet,vinot}@cena.fr

ABSTRACTInvolving graphic designers in the large-scale developmentof user interfaces requires tools that provide more graphicalflexibility and support efficient software processes. These re-quirements were analysed and used in the design of the TkZ-inc graphical library and the IntuiKit interface design envi-ronment. More flexibility is obtained through a wider paletteof visual techniques and support for iterative construction ofimages, composition and parametric displays. More efficientprocesses are obtained with the use of the SVG standard toimport graphics, support for linking graphics and behaviour,and a unifying model-driven architecture. We describe thecorresponding features of our tools, and show their use in thedevelopment of an application for airports. Benefits include awider access to high quality visual interfaces for specialisedapplications, and shorter prototyping and development cyclesfor multidisciplinary teams.

KEYWORDS: visual design, vector graphics, SVG, soft-ware architecture, GUI tools, model-driven architecture

CATEGORIES: H5.2 [Information Interfaces and presenta-tion] User Interfaces — GUI; D2.11 [Software engineering]Software Architectures.

INTRODUCTIONThe work of graphic designers in user interface projects waspopularised with the Web and desktop interfaces [14]. Withthe growing understanding that this work can improve users’performance and acceptance of new products, it is now soughtin the design of specialised user interfaces, from aircraft cock-pits to plant supervision systems. For instance, Figure 1 il-lustrates a graphic designer’s work for air traffic control.

This evolution raises an important engineering issue: howcan companies design and produce such design-intensive soft-ware at reasonable costs? For standalone applications withno complex functional core, tools such as Flash and Direc-tor allow designers to produce high quality products. But

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.UIST ’04, October 24–27, 2004, Santa Fe, New Mexico, USA.Copyright c© 2004 ACM 1-58113-957-8/04/0010. . . $5.00.

for more complex software, programmers and designers stillhave to choose between cost and flexibility. On the one hand,pre-designed widgets offered by interface builders are simpleto assemble. However programmers often produce poor lay-outs, and designers resent the low control over the look andinteraction style. On the other hand, graphical libraries offergreater flexibility: programmers can reproduce designs pro-vided by graphic designers within certain limits. But eventhen the limits are sometimes too strict for designers, and thecost of re-coding their work is too high to support both itera-tive design and a profitable industry.

Figure 1: Visual techniques, when used by graphicdesigners, can enrich the message conveyed.

Efficiently involving graphic designers in the production ofinteractive software calls for tools that meet the requirementsof both graphic design and software engineering. It requiresgraphical capabilities that match the working methods of de-signers. It also challenges the architecture of common userinterface tools and the notion of predefined widgets, whichwere introduced solely for programming purposes.

This article describes these two sets of issues and the solu-tions that we implemented in TkZinc and IntuiKit. TkZ-inc is a graphical library provided as a replacement for theCanvas component in the Tk toolkit [15]. It is available atwww.tkzinc.org. Co-designed with a graphic designer, itsfeatures are aimed at providing programmers with graphicalcapabilities and concepts that give them a common vocab-ulary with designers. IntuiKit is an interface design suiteaimed at making the prototyping and development of multi-modal user interfaces more accessible to industrial compa-nies. Its model-driven architecture [10] supports softwareengineering processes that involve iterative design and de-

267Volume 6, Issue 2

signers from multiple disciplines. In this paper, we focuson the features of IntuiKit that are aimed at graphic design-ers: the use of the Scalable Vector Graphics (SVG) standardas the graphical model of the environment, and an architec-ture that focuses on combining the graphical facet of inter-active components with the other facets that are producedby programmers. We then illustrate the use of IntuiKit on areal application, and discuss implementation details, relatedwork and the limitations and perspectives of our work. Allillustrations in this paper except diagrams are screen shots ofapplications created with TkZinc and IntuiKit.

MATCHING DESIGNERS’ WORKArt does not reproduce the visible; rather, it makes visible.

Paul Klee, 1920

Producing efficient visuals is not just a matter of good tasteand having tools with a rich graphical model. Graphic de-signers are trained to use methods evolved by artists overcenturies, whatever the media and tools. Understanding thosemethods helps selecting which features are most needed bydesigners and in what form. That is what guided the develop-ment of TkZinc: the goal was to make user interface designan experience similar to illustration design, notwithstandingthe fact that user interfaces are dynamic programs.

Being implemented as a Tk widget, TkZinc benefits fromstandard GUI mechanisms: windowing, event management,etc. The originality of TkZinc lies in its graphical primitives,chosen and organised as a mix of GUI programming librariesand drawing tools such as Adobe Photoshop and Illustrator.All other features, such as the programming interface or dis-tribution of events to graphical items, are modelled after theTk Canvas. The main graphical characteristics of TkZinc arevector-oriented graphics and visual effects, a 2D scene treewhere groups play an important role, and a graphical modelthat includes shape construction, gradients, clipping, trans-parency and shading. Some of these features are classicaland available in standards such as SVG, whereas others aremore radical or only available in 3D tools such as OpenGL.In this section we show how they were chosen, through theobservation of visual designers.

When working on a display or a user interface, a designertakes several types of information into account:

• the characteristics of the media to be used, for instance thesize and resolution of a screen;

• the data to be displayed, which is the main focus of soft-ware engineers;

• context information, provided by observation or more of-ten by marketing or internal communication departments:what is the message to be conveyed to users (as opposed tomere data display);

• efficiency considerations, based on psychological laws orknow-how.

Designing a display starts with a holistic approach: buildinga global picture that conveys the desired message. The goalis to evoke things known to the user and not to reproducethem, by tricking the spectator’s eye if necessary. Once thisglobal picture is defined in their mind, designers work onsolutions to reproduce it, using the available range of visual

elements and techniques. We now examine in more detailsthe techniques used, and how TkZinc supports them.

A palette of visual techniquesArtists and designers use many ways to convey the mean-ings they desire: contrasts, harmonies, light, etc. Combin-ing these signals allows them to communicate data, but alsocontext, culture or feelings, which contribute to the globalinterface usability and performance. The ideal designer’spalette is made of shapes (including type), colours, textures,light and rhythm. In practice, they have always had to ap-proximate it with available tools, whether physical or digital.For that reason, we observed that designers sometimes usegraphical tools in unexpected ways for programmers. Thishas guided the design of many parts of TkZinc.

Revisiting shapes Although our understanding of the worldis made of shapes, our perception of it is not. Usually, artistsdo not use “graphical objects” to produce shapes. Graphi-cal designers sometimes do, when they want to produce highcontrast and visual simplicity. But generally, they rather seethem as “construction lines” used to apply visual effects suchas color strokes, shadows or light which produce the desiredperception. For this reason, TkZinc offers shapes that can beused both as graphical objects or as geometrical templates.Furthermore, with graphical models based on objects, de-signers often lack intuitive notions such as holes and clip-ping. For that purpose, TkZinc provides two complementarysolutions. Shape operations make it possible to build shapesby assembling or subtracting other shapes, thus providing so-lutions for holes. Clipping is also available: a shape can beused as a clip mask for any graphical object or group.

Figure 2: Complex gradients used to create a rubber-like appearance (left) or to simulate a 3D design (right)

Revisiting colour Perception is based on contrast, and colourcontrast is the easiest to create. However, designers oftenwant to use gradients rather than mere colour juxtaposition.This is why TkZinc provides a rich gradient model, wheregradients can be defined from shapes. That includes linearand radial gradients, but also conical gradients, and gradi-ents computed from arbitrary paths. Figure 2 illustrates howgradients can be used to simulate shapes, perspective effects,and shadows. The button on the left is made of a circularblack arc filled with a conical gradient, then cloned, trans-lated and filled with a path gradient. Most shapes and shad-ows on the right are built with similar techniques.

Rhythm A complement to materials is visual rhythm, pro-vided for instance by repeated pencil lines or strokes on adrawing. Rhythm can be provided by filters applied to im-ages or by line textures (the richer version of dashed lines). Itcan also be produced by the cloning and layout of visual ob-jects along “networks”. TkZinc provides geometrical grids

268

that automatically clone and translate shapes and gradients.For instance, the top of figure 3 is made of one object built bymapping a square path on a grid, filled with a pixmap texture.

Figure 3: Rhythm can be obtained by repeatingshapes along networks and combining them

Light The contrast between light and shade is an importantway of signalling information as well as driving the orderin which users read displays. It is also a way to structurethe visual space, make it consistent and create 2-1/2D ef-fects. The designer in our group likes to manage light as a“skin” added on top of shapes, colours and textures. For thatpurpose, TkZinc offers a combination of groups and trans-parency, as do recent versions of Adobe Illustrator. Objectscan be hierarchically organised as groups, and a transparencycan be associated to any group, making it behave as a semi-transparent layer. Figure 1 makes heavy use of groups andtransparency.

Image constructionThe above techniques are basic blocks for designers’ work.However, some images cannot be obtained with a single op-eration. Oil painting, for instance, involves the progressiveconstruction of pictures with several layers of paint. Design-ers are used to building images in several successive opera-tions. Several features of TkZinc support this process, some-times perceived as sub-optimal by programmers: rather thanbeing pre-computed, images are recomputed during the in-teraction, which allows designers to manage interactive dis-plays while keeping the appropriate graphical control.

Figure 4: Hierarchical groups, simple transformations,clipping, shape operations and gradients help shapethe scene light and provide great expressiveness

Shape operations Building some shapes is easier throughthe combination of simpler shapes. For that purpose, TkZ-inc offers a set of algebraic operations on shapes: addition,subtraction, intersection. Shapes are successively built, filled

with textures or gradients, used to build other shapes, etc.Objects such as those shown on the left of Figure 4 can alsobe built by assembling triangles and shading the resulting tes-sellation.

Hierarchical structure The successive operations must oftenbe applied to sets of objects rather than just one object. Thecombination of groups, transparency and clip masks that wedescribed earlier plays an important role in this process. Thepicture on the right of Figure 4 is made of four shapes createdfrom seven circles and a gradient each, which are groupedand cloned. The cloned group is translated and scaled, thenclipped to produce the “magic lens” effect.

Attenuation and blending The iterative construction of im-ages involves sequences such as making an operation thensubtly attenuating it on some parts of the display, or makingtwo shapes and merging them with a superimposed layer thatblends the contours. Groups, gradients and transparency canbe used for that purpose. Filters such as blurring are not yetavailable in TkZinc but would also be desirable.

Parametric, interactive displaysMany of the above features are available, though often in dif-ferent forms, in drawing tools, graphical libraries, SVG play-ers, and in some UI toolkits. However, one should distin-guish between drawing and creating representations. Whatis needed to create high-quality user interfaces is a set of ob-jects that can be computed from application data, modifiedat run-time, and used to capture users’ action. Dynamicityis central, and TkZinc allows designers to create dynamic in-teractive objects, not just draw.

Extreme vector graphics There is a debate in the UI toolkitcommunity about bitmap and vector graphics: the formerprovide pixel-precise control to designers, whereas the lat-ter offer scalability. The design of TkZinc represents a rad-ical position in this debate: TkZinc aims at being a vector-only toolkit (with a slight concession to imported texturesand icons). Graphical objects are made of vectors of course,but gradients, clips, and grids are also based on geometri-cal vectors and can be changed at run-time. Similarly, allvertices of paths and curves can be accessed and changed in-dividually. No new feature is added to TkZinc if it cannotbe parameterised and related to the geometrical model. Thereason for this choice is interactivity: all graphical constructsmust be able to be bound to evolving data or interaction, andthus be recomputed at run-time.

Object selection Providing feedback often implies chang-ing the visual attributes of several objects at the same time.It is not always possible to put those objects in one group, be-cause they may be grouped with other objects for other pur-poses: clipping, blending, etc. Another way of addressingseveral objects must be provided. TkZinc provides a systemof tags inspired from Tk [15]: an object can have several tagsand it is possible to apply an operation to all the objects thatshare a given tag. We will see that in IntuiKit this system isextended to full XPath queries.

269Volume 6, Issue 2

GRAPHIC DESIGN AND SOFTWARE ENGINEERINGAugmenting the palette offered by GUI tools to designers isan important first step. However, in many contexts this isnot enough. First, this leads to a linear production process:graphic designers do their work, then programmers take over.Second, this involves a duplication of effort: programmershave to reproduce part or all of the graphics. We observedthat these two factors often prevent software developmentgroups from involving graphic designers, because of the re-lated cost and project management issues.

Wider acceptance of iterative design and graphic design re-quires solutions that preserve the ability to redesign graphics,and avoid duplication of effort. The direction we chose in thedesign of IntuiKit was to explore software engineering pro-cesses that give graphic designers a more central role in theproduction of software, while preserving the ability of pro-grammers to structure their code appropriately. Designersbecome software producers. Their artwork becomes a newtype of software component that can be managed in its ownway then merged with other components, just like softwarecomponents obtained from different source files have to bemerged by compilers and link editors in traditional softwareengineering.

IntuiKit is a software suite aimed at designing and proto-typing multimodal user interfaces, then transferring them toindustrial production. It is not centered around a modalitybut rather organised as an execution environment for modelsof software components, which makes it closer in many re-spects to a language interpretor than to a GUI toolkit. WithIntuiKit, a user interface is the result of the instantiation andcombination of several models. These are all provided by ex-tensions to the core software component model that is usedboth for structuring programs and merging models. The setof models used depends on the type of interface and the pre-ferred modelling approach: for instance, models of graph-ical objects and behaviours for GUIs, or speech and gram-mar rules for speech interfaces. These models can be builtand managed independently, each with the appropriate tools.They are then loaded by IntuiKit from XML files or instan-tiated through a programming interface, linked together withapplication code or non-modelled UI code, and “executed”.

In this section, we describe and illustrate the parts of IntuiKitthat are central to the involvement of designers in the soft-ware engineering process: the general architecture and soft-ware component model, the SVG graphical model, the dis-crete behaviour model, and how they are combined to pro-duce applications.

Architecture and componentsThe central structure of an IntuiKit application is a tree ofElements and Components. Elements are the nodes of thetree, and the basic blocks of all models: graphical objects,windows and behaviours are Elements. The execution of theapplication consists of a series of traversals of the tree, inwhich elements are activated. Initialisation, rendering andcontrol flows triggered by events are example of such traver-sals. A special kind of Element is the Reference, which al-lows shared objects in the tree. Elements can be declaredas models, making them insensitive to rendering traversals.

Elements can be cloned, thus giving IntuiKit some featuresof prototype languages. Elements can be loaded from XMLfiles. They can be parameterised through a property mech-anism: they export the names of their properties, which canbe set from Cascading Style Sheets (CSS) files [13]. Refer-ences can be defined using XPath expressions [20] to selecttheir target.

Components are Elements that contain children. They imple-ment encapsulation and parameterisation through a names-pace system. All element and property names exported bychildren are visible to their siblings. These names can in turnbe exported (with an optional renaming) by the parent Com-ponent to its own siblings and parents. Properties can also bemerged by the parent, which means that two sibling elementswill share the merged property. Control in IntuiKit relies ona classical event model, in which any Element and Compo-nent can emit events. Programmers can implement their ownComponents in native code. The IntuiKit tree can be under-stood as an extended scene graph. We prefer to interpret it asthe abstract tree of a language, because it is aimed at struc-turing software more than rendering graphics.

The core of IntuiKit only provides structuring and commu-nication mechanisms. All interactive capabilities are pro-vided by IntuiKit extensions; defining an extension consistsof defining new Element types and their semantics when ren-dered. Element types can be taken from existing standards, asare the SVG elements for graphics or VoiceXML for speech.They can also be defined specifically, as are the finite statemachines that are used below for defining the basic discretebehaviour of interactive components.

Figure 5: Henry-the-Frenchie

SVG: turning artwork into softwareSVG, a recommendation by the World Wide Web consor-tium, is an XML standard for exchanging vector graphics.Its object-oriented graphical model covers an important partof the requirements described earlier in this article, whichmakes it a good candidate for storing user interface graph-ics. SVG is also convenient as a graphical file format: it canbe generated from graphic design tools such as Adobe Illus-trator or Corel Draw. For these reasons, and despite somedisparities with the features described earlier, the GUI mod-ule of IntuiKit implements the SVG model, using TkZinc asits graphical rendering engine. SVG elements are consideredas IntuiKit Elements, and SVG groups as Components. In-tuiKit offers two equivalent ways for manipulating graphics:by loading SVG files or through a programming interface.Using files, one can manage graphics as an independent part

270

of the software, produced for instance with Adobe Illustratorby saving the artwork in SVG format.

The two fragments of Perl code below show how the GUImodule of IntuiKit is used. The first example uses the pro-gramming interface and opens a window that displays sometext. It illustrates how the position of Elements in the treeinfluences the result just like in a scene graph: the window(Frame), font and text are added to the root Component insuch an order that the text appears with the appropriate fontin the window.

$root = new Component;

new Frame ( -parent => $root );

new Font ( -parent => $root, -family => ’Helvetica’ );

new Text ( -parent => $root, -text => ’EAT!!!’ );

$root->run;

The second example illustrates the use of an SVG file. It actsas a basic SVG player, by adding a window and an Elementto a root Component. This is illustrated in Figure 5 with theSVG file from a very simple application that we will use inthe rest of this section: a digital pet inspired from a popularelectronic toy. The pet, named Henry-the-Frenchie, is totallyuseless and has a simple behaviour: it grumbles when hungryand can be fed by pressing a button.

$root = new Component;

new Frame ( -parent => $root );

$henry = load Element ( -parent => $root,

-file => ’henry.svg’ );

$root->run;

Considering structured graphics files as a software compo-nent allows designers to deliver successive versions and thusto work in parallel with programmers. However, conven-tions must be established. IntuiKit uses Element names forthat purpose: just like function or component names mustbe determined in advance between programmers, SVG ele-ment names must be agreed upon by designers and program-mers. As an example, Figure 6 shows an extract of the AdobeIllustrator palette that represents the artwork for Henry-the-Frenchie. These layers and groups are saved in SVG as a treeof groups with the names that were set in the palette. Conse-quently, the palette reflects the contract passed between thedesigner and the programmer.

Figure 6: Structure of the SVG file for Henry

Bringing graphics to lifeProgrammers must be able to capture events on graphical ob-jects and change their appearance upon user events or mes-sages from the functional core. For instance, Figure 7 illus-trates event flows in the Henry-the-Frenchie application. The“start eating” and “stop eating” events are generated whenthe user depresses and releases the “eat” button. A simplifiedfunctional core (FC) manages an internal numerical valuethat grows while the button is depressed, then decreases peri-odically. The FC emits “increase” and “decrease” events thatcorrespond to Henry’s status changing from starved to sati-ated or stuffed and back. The interactive component reactsupon all events to provide visual feedback.

Describing behaviour is a different facet than graphics. Thecore module of IntuiKit provides an event system based ona classical binding mechanism. A binding associates an ac-tion with an event source (clock, graphical object, etc) and aspecification (button click, key press, etc); a binding can bedynamically enabled or disabled. The GUI module that im-plements the SVG model defines graphical objects as eventsources like the Tk Canvas: when the user clicks, the firstvisible object under the mouse with an active binding hasits binding triggered. Bindings on groups can be specifiedas “atomic” so that events are detected in all elements of agroup. This binding system provides the foundation for pro-gramming behaviour as a set of callback functions, or forusing more elaborate models.

increase

start eating

stop eatingUser Input Interactive

Component

FCstop eating

decreasestart eating

Figure 7: Event flows in Henry

The literature provides various models of discrete or contin-uous behaviour in user interfaces, such as StateCharts, finitestate machines, Petri nets, data flows, constraints, etc. Anyof these models could be offered as a set of Elements in In-tuiKit, provided that the appropriate semantics are definedand implemented (some of these models may require im-provements to the core of IntuiKit). For discrete behaviours,IntuiKit currently implements finite state machines (FSMs).Although its limits are well known, this model is rich enoughto define basic interactive components, and it illustrates howsuch a model can be combined with graphical Elements. Con-tinuous behaviours such as scaling or translation are alsoavailable; they rely on the ability of the GUI module to changethe attributes of SVG elements once they are instantiated.

Finite state machines: IntuiKit defines FSMs as a set ofstates and a set of transitions labelled with event specifica-tions and actions. When in a state, a binding is activatedfor each outgoing transition. Actions include traditional call-backs and event emission. Event specifications can refer toall Element names that are visible from the FSM (the namesexported by its siblings). This makes it possible to asso-ciate behaviours to events occurring on graphical Elementsdefined in the same Component.

271Volume 6, Issue 2

IntuiKit FSMs are local to their parent Component. Theycan be used to store its state and to emit events or triggercallbacks when certain sequences of events occur. They canalso be used for basic graphical feedback by triggering thedisplay and hiding of graphical Elements that are associatedto its states. This takes advantage of a method often usedby graphical designers who use Adobe Photoshop to buildbehaviours in Web pages: they store the different states oftheir objects into different layers and manually simulate thetransitions by turning the visibility flag of the different layerson and off. IntuiKit offers this possibility by using propertymerging and the Switch Element.

Figure 8: The FSM-Switch pair for Henry’s stomach

Switches: In SVG, a Switch is a special group that rendersonly one of its children depending on the value of a variableof the run-time context, such as the preferred user language.IntuiKit extends the semantics of the Switch, by defining animplicit property named “branch” that controls which childof the Switch will be rendered. If the Switch is present in thesame Component as a FSM, one can synchronise the Switchand the FSM by merging the “branch” property of the Switchand the “state” property of the FSM: when the FSM changesstate, it triggers a partial traversal of the tree, and the Switchis rendered according to its new active branch.

Figure 9: The complete IntuiKit tree for Henry

Figure 8 shows the FSM-Switch pair used for Henry’s stom-ach. The FSM receives “increase” and “decrease” eventsfrom the FC. It is associated with a Switch that activates thegroup containing the appropriate picture for Henry. Figure 9gives a more complete view of the tree, and shows anotheraspect of combining models. Switches control the renderingof all their children, including FSMs. When a FSM is notrendered by its parent Switch, its event bindings are disabledand its state memorised. This provides control similar to thatof hierarchical FSMs, and it is used at two levels of depth for

Henry: it can be Idle or Eating (first FSM), and Starved, Sati-ated or Stuffed (second FSM) when Idle. The code fragmentbelow illustrates the creation of the tree with XPath refer-ences, FSMs, including event specification, and the mergingof properties which synchronises a FSM and a Switch.

$eat_model = $svg->find( -ref => ’.//Eating’ );

$eat_model->clone( -parent => $Henry );

...

$switch = new Switch( -parent => $Henry,

-branches => { ’eating’ => ’../Eating’,

...

$fsm = new FSM( -parent => $Henry,

-states => [ ’eating’, ’idle’ ],

-transitions => {

{-from => ’idle’, -to => ’eating’,

-on => [$eat_button, ’StartEating’]},

...

$Henry->merge( -names => [$fsm->state, $switch->branch] );

Adjusting parametersUser interface environments must support the managementof parameters: colours, text labels, etc. In IntuiKit, this isobtained with CSS: all properties defined by Elements canbe set from a style sheet file. For instance, let us assume thatthe Functional Core (FC) of Henry is implemented in Perl asa Component, and that the values of its “appetite” and “di-gestion” control its internal algorithms. The correspondingproperties would then be defined as follows:

package FC;

use Component;

sub new {

my $self = new Component;

$self->define( -name => ’appetite’, -default => ’10’);

$self->define( -name => ’digestion’, -default => ’5’);

...

Then, the functional component could be created as follows:

$root = new Component ( -name => ’Henry’ );

$fc = new FC ( -parent => $root, -name => ’fc’ );

And finally, the properties could be changed with the follow-ing style sheet file:

Henry/fc {

appetite: 50;

digestion: 44;

}

IMPLEMENTATIONTkZinc is a free software graphical library distributed byCENA since 1998. Aimed at prototyping, it is built as awidget in the Tcl-Tk environment, and can be used fromthe Tcl, Python and Perl scripting languages. Its core struc-ture is coded in C and originally based on the X WindowSystem. Many advanced visual features of TkZinc requireOpenGL. TkZinc runs on Windows, Linux and Mac OSX.Initially developed for safety-critical displays, TkZinc hasgood CPU performance and reliability: earlier versions of

272

TkZinc were used in operational air traffic control worksta-tions from 1998 to 2003 for interactive displays composedof thousands of graphical objects. Benchmarking against theJava-based Batik SVG player revealed that TkZinc was morethan 10 times faster when loading and displaying a figuremade of 2600 curves amounting to 84000 vertices.

IntuiKit is developed and distributed by IntuiLab since 2003.It offers a prototyping environment programmed and accessi-ble in Perl, that has been used to develop car displays, touch-screen-based workstations for various domains, as well asprototype multimodal applications. It runs on Linux andWindows. A C port accessible in C++ and Java and aimedat small devices and production code is under development.

EXAMPLE APPLICATIONIntuiKit is used to develop prototypes or pre-operational prod-ucts in various domains: automotive, aerospace, manufactur-ing, telecommunications and defence. This has given us theopportunity to test the concepts described in this article onreal applications. Four graphic designers were involved inthe projects, one at a time. Some had no previous experienceof working with programmers. In all cases the collabora-tion was remote, meetings being reserved for participatorydesign sessions. These experiences allowed us to refine theproposed process and communication conventions betweenproject managers, graphic designers and programmers. Theyalso provided data to assess the gains brought by the chosenarchitecture in terms of schedules and effort.

We now describe the development of a departure managerfor airports, which was designed and developed over a pe-riod of a few weeks in late 2003. It is a good example of howprofessional applications can benefit from giving designersmore expressive power, and of the work process supportedby our choice of software architecture. A company had de-veloped algorithms to optimise the sequence of departuresfor controllers who guide taxiing aircraft. Air traffic con-trollers are known to be very demanding professional users,and the company wanted to embed their algorithms in touch-screen workstations that would both provide excellent usabil-ity and seduce users and decision makers. The applicationwas meant for pre-operational tests, and consequently had tooffer full functionality. The company also had a very tightschedule because they wanted to exhibit the product at a pro-fessional convention so as to gain customers. This deadlineobviously played a key role in the organisation of the project.

The project team was composed of two programmers (onebeing the lead interface designer and the other a domain ex-pert), and a graphic designer. The project started with a dis-count participatory design session that produced a paper pro-totype for every interface to be built in the project. Figure 10shows one of the prototypes. This prototype served as thereference for all further developments on the interface, inseveral ways. First, the layout of the different parts of theinterface was the result of collective work and served as thebasis for future composition work by the graphic designer.Second, all parts of the prototype were given a name: staticparts (“printer”, “column”, “timeline”, etc) as well as a tem-plate name for dynamic parts (“strip”, “plan”). The namesserved as a basis for informal communication between par-

Figure 10: The paper prototype that served as a refer-ence for group work

ticipants, who never met again until the end of the project.Third, immediately after the session the lead interface de-signer decomposed the prototype into a tree of components,starting with the top-level parts. This tree represented the ar-chitecture of the application, both in terms of software com-ponents and graphical components. She used the tree as acollective contract and interface between project actors, es-pecially between UI programmers and the graphic designer.

From then on, the programmers and the graphic designerworked independently, contacts being limited to clarificationquestions or visual design proposals and feedback. The de-signer started working on the general impression he wantedto convey, and tested ambiances, colours, harmonies, tex-tures, etc. Figure 11 gives a sample of his work.

Figure 11: The designer’s work on “the global picture”

Meanwhile, programmers used the reference component treeto code the application. For each component they definedthe behaviours, the computations, and the connection to thefunctional core. They also created basic graphics, whichwere needed for testing the code. They used a professionaldrawing tool to produce some of the graphics, others beingcoded with the IntuiKit Perl API. The result of their work isshown in Figure 12. The application was of very limited vi-sual quality, but sufficient to test its usability as well as theconnection to the functional core, implemented as a server.

Three to four weeks before the deadline, the designer had fin-ished maturing the design and preparing his elements: fonts,patterns, etc. He was then ready to produce graphical el-

273Volume 6, Issue 2

Figure 12: The application with programmer-made graphics

ements to replace the placeholders made by programmers.Having a computer equipped with IntuiKit and being regu-larly sent new versions of the application, he was able to testthe elements in context himself: for that, he just had to dropSVG files in the appropriate places. When satisfied with theresult, he sent the SVG files to the programmers by email.Figure 13 shows two SVG designs that are perfectly equiva-lent from the application’s point of view, though not exactlyequivalent for the user.

Figure 13: Detail: two skins for the “strips”

Finally, after a period of design, test, feedback, redesign, andintegration, the final graphics were produced a few days be-fore the deadline. The very low cost of integrating graphicsas well as the ability to work in parallel allowed for thesevery late iterations while programmers were busy testing anddebugging the application. This process allowed every actorof the project to manage time in their own way to work untilthe last limit, without excessive tension. The final result isshown in Figure 14. As a final note, the product was a greatsuccess with potential users and customers at the exhibition,and its interface was perceived by all as an important com-petitive advantage. This is one more empirical proof of thepotential of graphic design for professional systems and theimportance of supporting it.

DISCUSSION AND PERSPECTIVESPractical experience with TkZinc and IntuiKit such as theone we just described has evidenced both benefits and re-maining limitations. The most obvious benefit of using richgraphical models is the visual quality of the resulting userinterfaces, when designed by professionals. But we con-

sider the efficiency gains as equally important. The projectdescribed above has allowed us to compare figures with aproject of similar size and complexity executed by the sameteam a few months earlier without Intuikit. The earlier projectinvolved the same effort by the graphic designer (15 days)and the same amount of non-graphical UI code (15 kloc). Itwas built with a more traditional process: the designer pro-duced visual elements that the programmers reproduced withtheir programming language (Perl, used in object-orientedstyle). We have observed three major improvements in theexecution of the second project with IntuiKit:

• a 20-30% reduction in programming effort, obtained byavoiding the re-coding of graphics;

• a 50-70% reduction in overall project length, obtained withthe parallel production process, even though a pipe-lineprocess was set up in the earlier project to incorporate vi-sual elements as they were produced;

• a reduction in coordination costs that we did not measurebut that we estimate at 50-70% in terms of number andduration of telephone calls.

We also have identified several limitations and possible im-provements:

• Freeing graphic designers from certain constraints has amajor impact on performance: they create much more com-plex visual representations. Between the two projects men-tioned above, memory use has more than doubled (from30 Mb to 70 Mb with IntuiKit). Similarly, graphical per-formance becomes an issue again, even with recent GPUs.Animations in the application illustrated in Figure 14 arenot smooth enough with low-end GPUs.

• The standard SVG format does not support all types ofblending between layers, neither does TkZinc. Some de-signers consider that as a constraint, since they use blend-ing in their iterative construction of visual effects. An op-tion might be to support SVG extensions provided by somegraphical tools.

• IntuiKit does not yet support SVG filters, which could beuseful to designers especially in terms of rhythm and light.However, performance issues would have to be monitored.

• Some of the features of TkZinc are lost when using In-tuiKit and SVG. That forces a trade off between devel-opment cost and visual richness. Evolutions of the SVGstandard could help avoid that trade off.

• The use of OpenGL in TkZinc provides expressiveness andperformance. However, extra care had to be taken whenimplementing features with OpenGL and there still are afew conceptual flaws caused by it. OpenGL is not primar-ily aimed at 2D graphics design, and some operations leadto inappropriate results as far as visual designers are con-cerned: lack of pixel-precise control in certain cases, andunnatural results for certain colour combinations.

IntuiKit was tested successfully on projects where the ini-tial iterative design was done with low-fidelity prototypes,and production started once the general structure of the in-terface was decided upon. It has not been tested for earlydesign phases. Regarding the response of teams, experienceshowed that many programmers are not yet used to program-ming user interfaces as components communicating throughevents, and require some training. Once familiar with this no-

274

Figure 14: The final application, after full integration

tion, programmers exposed to IntuiKit seem to have no prob-lem with the manipulation of FSMs. Designers enjoyed theprocess, and some wanted to go further. The supported pro-cess confines them to the role of providing visual elements,when they could define part of the behaviour of visual ele-ments: visual styles that would define what a progress barwould look like for the whole range of values, for instance.We are considering the use of rules or constraints [18], aswell as data-flow techniques inspired from [4], [9] or [5].

Future work also includes the implementation of new mod-els. For GUIs, that would include layout, non-linear geomet-ric transformations, and animation. Finally, access for non-programmers would require tools for building models, whenno commercial tools are already available. This includes agraphical editor for building behaviours and associating themto graphics. This could also include other techniques fromthe literature to build or infer models, such as programmingby demonstration.

RELATED WORKAlthough mostly aimed at 3D rendering, the wide availabilityof OpenGL-capable graphics cards stimulates the introduc-tion of richer 2D tools. For instance, the new Java 1.5 addsOpenGL-based rendering for Java2D. Similarly, the scenegraph of 3D toolkits such as Open Inventor [21] has also in-spired the design of toolkits for programming post-WIMPuser interfaces [2, 1, 11]. The focus of these toolkits is onproviding extendable architectures for developing novel in-teraction and visualisation techniques. Ubit for instance isbased on a scene graph model for combining visual elements

with behavioral elements and layout elements [11]. Theseelements are defined in a declarative C++ style, and thus arenot accessible to graphic designers. CPN2000 is a graphi-cal Petri net editor [1]. It has been built with a program-ming toolkit that supports transparency and non-rectangularwindows rendered with OpenGL, and uses SVG to displayicons. Many SVG renderers such as the Batik toolkit [19]are available, as well as SVG extensions to existing toolkits.Most of them are limited to displaying graphics and do notprovide programmers with access to graphical objects so asto manage interaction. Similarly, little consideration is givento work processes and the role of graphic designers.

Previous work on involving graphic designers in the produc-tion of user interfaces can be analysed according to the levelof flexibility given to designers. At one end are designer-made widgets that can be reused by programmers. In themiddle are tools that incorporate designers’ know-how forproducing semi-automated designs: the Kandinsky system [6]provides programmers with templates they can transform forpresenting data; the Kinetic typography engine [12] encap-sulates animation techniques from cartoons. At the otherend, Hudson et al [7, 8] propose solutions for splitting wid-gets into several pixmap zones. That allows programmers tochoose among the library of pixmaps proposed by designers,while remaining able to resize widgets at will. By using vec-tor graphics, the IntuiKit GUI module avoids the problem ofresizing. In addition, by not relying on a predefined set ofwidgets, it gives the designers total control over the look andbehaviour of interactors, but does not support the creation

275Volume 6, Issue 2

of themes. With IntuiKit, designers produce the graphicalpart of the interface, rather than providing styles for graph-ical objects built by programmers. The two approaches canbe combined.

Regarding software architecture, IntuiKit can be comparedto the model-driven tools that use the XML format [17]. Theclosest are the XUL [3] and XAML widget sets, which allowvendors to provide variants of widgets as long as they im-plement the specification. IntuiKit is more oriented towardspost-WIMP interfaces and allows one to freely manipulategraphical objects and behaviours, whereas XAML widgetsare provided as compiled C# code. Current efforts by theW3C are aimed at extending SVG with other markup lan-guages such as XForms [16] for describing behaviour, butput less stress than IntuiKit on separating concerns.

CONCLUSIONWe have described the design and features of the IntuiKit in-terface design suite and its graphical engine TkZinc aimed atsupporting designers’ needs when developing user interfaces.The features of TkZinc allow programmers to better matchthe work of designers. The architecture of IntuiKit, based onthe combination of models, allows teams to manage graph-ics separately from behaviour and other software concernsand then merge them into a structured tree of components. Itorganises the collaborative work of designers and program-mers. The SVG format, though sometimes restrictive, is keyto this process because it allows designers to use their owntools to produce parts of the interface. The architecture ofIntuiKit helps bridge the gap between iterative design andsoftware engineering cycles, and is ready for integrating newmodels and making them available to development groups.

ACKNOWLEDGEMENTSFrédéric Lepied (now with Mandrakesoft), Dominique Ruizand Stéphane Valès contributed to the implementation of TkZ-inc and IntuiKit. Yves Rinato (Intactile Design) designed thedeparture manager, which is shown with the kind permissionof Sofréavia. Michel Beaudouin-Lafon was of great help inimproving earlier versions of this paper.

REFERENCES1. M. Beaudouin-Lafon and H. M. Lassen. The archi-

tecture and implementation of CPN2000, a post-WIMPgraphical application. In Proceedings of the ACM UIST,pages 181–190. ACM Press, 2000.

2. B. Bederson, J. Meyer, and L. Good. Jazz: an extensi-ble zoomable user interface graphics toolkit in Java. InProceedings of the ACM UIST, pages 171–180, 2000.

3. D. Boswell, B. King, I. Oeschger, P. Collins, andE. Murphy. Creating Applications with Mozilla.O’Reilly, Sept. 2002.

4. S. Chatty. Extending a graphical toolkit for two-handedinteraction. In Proceedings of the ACM UIST, pages195–204. Addison-Wesley, Nov. 1994.

5. P. Dragicevic and J.-D. Fekete. Input device selectionand interaction configuration with icon. In Proceed-

ings of HCI-IHM 2001, pages 543–448. Springer Ver-lag, Sept. 2001.

6. J. Fogarty, J. Forlizzi, and S. E. Hudson. Aesthetic in-formation collages: generating decorative displays thatcontain information. In Proceedings of the ACM UIST,pages 141–150, 2001.

7. S. E. Hudson and I. Smith. Supporting dynamic down-loadable appearances in an extensible user interfacetoolkit. In Proceedings of the ACM UIST, pages 159–168, 1997.

8. S. E. Hudson and K. Tanaka. Providing visually richresizable images for user interface components. In Pro-ceedings of the ACM UIST, pages 227–235, 2000.

9. R. Jacob, L. Deligiannidis, and S. Morrison. A softwaremodel and specification language for non-WIMP userinterfaces. ACM Transactions on Computer-Human In-teraction, 6(1):1–46, 1999.

10. J. W. Jespersen and J. Linvald. Investigating user in-terface engineering in the model driven architecture. InProceedings of the Interact 2003 Workshop on SoftwareEngineering and HCI. IFIP Press, Sept. 2003.

11. E. Lecolinet. A molecular architecture for creating ad-vanced GUIs. In Proceedings of the ACM UIST, pages135–144, 2003.

12. J. C. Lee, J. Forlizzi, and S. E. Hudson. The kinetictypography engine: an extensible system for animatingexpressive text. In Proceedings of the ACM UIST, pages81–90, Oct. 2002.

13. H. W. Lie and B. Bos. Cascading style sheets, design-ing for the Web. Addison-Wesley, 1999.

14. K. Mullet and D. Sano. Designing Visual Interfaces.Prentice Hall, 1995.

15. J. K. Ousterhout. Tcl and the Tk toolkit. Addison-Wesley, 1994.

16. A. Quint. SVG and XForms: A primer. IBM develop-erWorks, Nov. 2003.

17. N. Souchon and J. Vanderdonckt. A review of XML-compliant user interface description languages. In Pro-ceedings of DSV-IS 2003, pages 377–391. Springer-Verlag, 2003.

18. P. Szekely and B. Myers. A user interface toolkit basedon graphical objects and constraints. In Proceedings ofOOPSLA, pages 36–45. ACM Press, 1988.

19. The Apache XML project. Batik SVG Toolkit.http://xml.apache.org/batik/, 2004.

20. W3C Recommendation 16 November 1999.XML Path Language (XPath) Version 1.0.http://www.w3.org/TR/xpath, 1999.

21. J. Wernecke. The Inventor Mentor, programmingobject-oriented 3D graphics with Open Inventor, Re-lease 2. Addison-Wesley, 1994.

276