How to Survive Multi-Device User Interface Development with UIML

Jo Vermeulen and Jan Meskens

Hasselt University – tUL – IBBTExpertise Centre for Digital Media

Belgium

{jo.vermeulen,jan.meskens}@uhasselt.be

A little bit about us …

A little bit about us …

About 80 researchers …

Specializing in …

A little bit about us …

About 80 researchers …

Specializing in …

The UIML Crew

Kris Luyten Jo Vermeulen Jan Meskens

Professor PhD student PhD student

• PhD about XML UIDLs• developed Uiml.net• several research projects• > 25 related publications• numerous students

• extended Uiml.net• MSc thesis• 2 applied research proj.• 6 related publications• 6 students

• extended Uiml.net• developed Gummy• BSc & MSc thesis• basic research project• 5 related publications• 1 student

Since 2002 Since 2004 Since 2006

What is UIML?

A description of a user interface

UIDLs in practice

UIML is differentUIML is different

UIML is different

• High level

• Generic

– No specific tags (e.g. <button>)

• Meta: build your own User Interface Language

– cfr. XML

What does UIML look like?

What does UIML look like?

<uiml><interface><structure><part>...</part>

</structure><style><property>...</property>

</style><behavior><rule>...</rule>

</behavior></interface><peers>...</peers>

</uiml>

What does UIML look like?

A bit of history

UIML: a pioneering UIDL

Professor Marc Abrams (Virginia Tech, USA)

• 20 years experience with building GUIs

• Frustrated with then-current methods

• UIML 1.0 spec and Java renderer

• Founded Harmonia to provide tools & services for UIML

1997: UIML is born.

UIML: a pioneering UIDL

• 1998

– PhD dissertation by Constantinos Phanouriou.

– Corresponding UIML 2.0 spec

• 1999:

– Windows CE renderer

– Demo at WWW6 conference / W3C DI workshop

• 2000:

– HTML, WML, VoiceXML renderers

1998-2000: updated specification & tool support.

UIML: a pioneering UIDL

• 2001

– UIML 3.0 spec

– LiquidUI tool (Harmonia)

– UIML Europe 2001 conference (Paris, France)

• 2002:

– UIML Technical Committee starts

• 2003:

– Kris Luyten creates open source Uiml.net renderer

• 2004:

– UIML 3.1 specification

2001-2004: UIML comes of age

The company behind UIML

Harmonia, Inc.

• contracts with US DoD

• e.g. UIs for missile control

UIML as abstraction

UIML as abstraction

Hardware

OS

UI toolkits Logic

UIML

Same device, different toolkit

Device A

…

…

Toolkit 1 …

UIML

Device A

…

…

Toolkit 2 …

UIML

Different device/OS/toolkit

…

…

… …

UIML

Device A Device B

…

…

… …

UIML

Multi-device/toolkit examples

Extra component necessary

Hardware

OS

UI toolkits Logic

UIML

UIML renderer

UIML Renderer

• Runtime environment

• Allows for:

– rapid prototyping

– dynamic changes at runtime

Our UIML Renderers

• .NET

– Works on most .NET-enabled platforms:

• Mono (*NIX)

• Microsoft .NET

• Compact .NET

• Java– Runs on various Java VMs and APIs, e.g.

• Java ME

• IBM J9

• MHP Xlet

UIML research/teaching

Dynamic mobile guide UIs

Context-aware UIs

Adapt according to the context-of-use (e.g. location, user profile)

Dynamic changes at runtime

OSGi + UIML

UIML Activator Bundlecontrols rendering core and connects with other OSGI components

Distributed Applications

• MSc thesis Neal Robben

• Distributed user interfaces with Uiml.net

• Synchronization between different UI instances

UIML and distributed Applications

UIML and X + V

• MSc thesis Rob Van Roey

• X+V backend for UIML

• multimodal user interfaces

– GUI interaction (XHTML)

– speech (VoiceXML)

UIML and X + V

UIML on CarMelody platform

• Thesis Pieter Gevers (industrial engineering)

• Prototyping UI with Uiml.net that invokes hardware APIs of Philips car radio

• Philips PNX0106 running

Montavista Linux

UIML on CarMelody platform

Disadvantages of UIML

• Lacks mature/available tools

• Lacks community

• Lacks documentation

• Lacks proof that it “just works” in industrial/commercial settings

What about performance?

Smart optimization

• Rendering certainly slower

• But on the other hand, …

• The UI description provides more data, allowing for smarter optimizations

Case study

• iDTV application, several linked UIML “pages”

• Painfully slow (7 seconds to render a page)

• Serious performance boost after optimization using pre-caching and MPEG-2 frames

Performance comparison

7

0,35

0

1

2

3

4

5

6

7

8

Rendering time per page (seconds)

Unoptimized

Pre-caching

Intermediate UIML

Motivation

• UIML language is platform independent

• Renderers have to be created

– For different computing platforms

– Different programming languages

Java Swing

Swing Backend

UIML for Java

Hardware

OS

Java AWT Logic

UIML

UIML Java renderer

AWT Backend

• Containment

• UI type conversion

UIML for .NET

Hardware

OS

GTK# Logic

UIML

UIML .NET renderer

GTK# Backend

SWF

SWF Backend

High maintenance and development cost

Hardware

OS

Toolkit Logic

UIML

UIML .NET renderer

Toolkit Backend

Hardware

OS

ToolkitLogic

UIML

UIML Java renderer

Toolkit Backend

Flexible Intermediate rendering

Hardware

OS

Toolkit Logic

UIML

UIML .NET renderer

Toolkit Backend

Hardware

OS

Toolkit Logic

UIML

UIML Java renderer

Toolkit Backend

iRenderer iRenderer

Intermediate Renderer

Structure

Style

Behavior

Vocabulary

UIML iUIML

Concrete Widget

PropertiesCallbacks

Concrete Widget

PropertiesCallbacks

Concrete Widget

PropertiesCallbacks

Concrete Widget

PropertiesCallbacks

Concrete Widget

PropertiesCallbacks

IntermediateRenderer

Example UIML

<d-class id=« frame » maps-to=« System.Windows.Forms.GroupBox »>…

</d-class>

Corresponding iUIML

Intermediate UIML

• JSON language (http://json.org)

• All UIML abstractions are resolved

– Concrete widget names

– Concrete property names

– Concrete event names

• iUIML is easy to translate into a running UI

• We built an Adobe Flex renderer in 3 days…

Adobe Flex Renderer

Design Tools

Motivation

• Learning UIML takes time

– Especially for non-technical designers

• Polishing a UI in UIML is difficult

– Requires switching between edit and run mode

Motivation

• Creating one UI for a range of devices is challenging

• Automatic transformations often not aesthetically pleasing

• Manual adjustment is needed

Design Tool: Gummy

Gummy

• Platform independent GUI builder

• Features

– Design GUIs graphically

– Export GUIs as UIML

– Transform GUIs from one platform to another (based on UIML)

– Load a platform-specific design workspace dynamically

Dynamic Design Workspace Loading

Gummy Architecture

DVB-MHP renderer

PDA renderer

Desktop renderer

Desktop Proxy

PDA proxy

DVB-MHP proxy

<uiml></uiml>

Desktop

W2P

Problem in Mobile UI Design

Designer Device

Target Device

Spacing and Positioning

• Foto

Touch Screen Characteristics

Touch Screen Sensitivity?

Occluded Areas?

Interaction Techniques and Emulators

Our Solution: Gummy-Live

“Mirror design changes to the mobile target device in real time”

Gummy-Live

Conclusion

Renderer Tool Support Designer Support