SKETCH-BASED USER SKETCH-BASED USER INTERFACE STUDYINTERFACE STUDY

Presented ByPresented By

Jin XiangyuJin Xiangyu

  

Department of Computer Science and TechnologyNanjing University

June 2002

PART I: PART I: INTRODUCTIONINTRODUCTION

The rise of the research issue of Human-Computer Interaction (HCI)

The idea of this revolution is to bend computers to people’s way of interactiThe idea of this revolution is to bend computers to people’s way of interacting, not the other way around (Landay 2001) ng, not the other way around (Landay 2001)

Human and computer, which one should be the center of computer assisted tasks?

Computer-OrientedComputer-Oriented

Human-OrientedHuman-Oriented

1.1. A Revolution is Undertaking1.1. A Revolution is Undertaking

Computers are becoming more and more powerful and easily available today

1.2. Why Sketch-based User Interface?1.2. Why Sketch-based User Interface?To write down designer’s improvisatory ideas by diagrams is very

important for creative tasks.

(3) Unsuitable for handled devices

No area to accommodate so many stencils and buttons.

Traditional menu/toolbar button-based user interface Demo

(1) Inefficient

A three-step process.

(2) Unnatural

Leaving a sketch uninterrupted, or at least in its rough state, is key to preserving this fluidity (Hearst 1998).

The Solution is “By Sketch”.

Sketching with a pen is a mode of informal, perceptual interaction that has been shown to be especially valuable for creative design tasks (Gross 1996) .

1.3. Research focus: What Kind of Sketch-1.3. Research focus: What Kind of Sketch-based User Interface We are Interested in?based User Interface We are Interested in?

On-line VS Off-line

Mode Input Process Segmentation Vectorization Stroke

On-line f(t): R+ R×R Interactive Easy Not required Cursive

Off-line 2D-Boolean Matrix One-step Difficult Required Formal

Three application level

Sketchy symbol recognitionStroke-number and stroke order free

Not very strictly defined

2

Sketch-based image retrieval

Handwriting recognition

Example

Totally free

Generally agreed among users

Drawing ApproachSymbol SetLevel

Undefined

Strictly defined

3

1

Our research focus

1.4. Three Designing Principles: 1.4. Three Designing Principles: Humanistic, Intelligent, IndividualizedHumanistic, Intelligent, Individualized

How to make the UI to be humanistic?

An graphics inputting user scenario is proposed, which employs an interactive sketching-recognition-rectification process in one-fluent-step.

How to make the UI to be intelligent?

On-line graphics recognition is employed to predict user’s original intention.

How to make the UI to be individualized?

SVM-based incremental learning is employed to adapt different user in shape classification.

These three characteristics are harmoniously combined in our prototype system

—Smart Sketchpad.

PART II: PART II: HIGHLIGHTSHIGHLIGHTS

2.1. Critical Technique 1:2.1. Critical Technique 1:User Scenario for Graphics Inputting User Scenario for Graphics Inputting

Employing one interactive, fluent sketching-recognition-rectification process instead of three split ones.

Step 1: Sketching Recognize and Regularize

The suggested candidate objects

Step 3: Clicking on the intended object and replace the strokes with the very object with proper parameters

Step 2: Sketching Recognize and Regularize

2.2. Critical Technique 2:2.2. Critical Technique 2:On-line Graphics Recognition On-line Graphics Recognition

Strokes Primitive Shape Composite Graphic Object

Primitive Shape Classification and

Regularization

Primitive Shape Classification and

Regularization

Composite Shape Recognition

Composite Shape Recognition

Sketching (user) : Decompose

Composite Graphic Object

Recognition (computer) : Assemble

Primitive ShapeStrokes

2.2.1. Primitive Shape Classification 2.2.1. Primitive Shape Classification and Regularizationand Regularization

Strokes

Primitive Shape Classification & Regularization

Primitive Shapes

The input stroke

Preprocessing Shape Classification

Quadrangle

Shape Fitting

The fitted shape

Shape Regularization

The regularized shape

The User Intended Shape

The User Intended Shape

The User Sketchy Shape

The User Sketchy Shape

By Vertex Combination

By Vertex Combination

Experimental Experimental ResultsResults

0.2 0.4 0.6 0.8 1.0 1.2 1.482

84

86

88

90

92

94

0.2 0.4 0.6 0.8 1.0 1.2 1.482

84

86

88

90

92

94

User 2R

eco

gn

itio

n P

reci

sio

n (

%)

Threshold of Force

User 1

Shape Classification Precision for 1367 samples

User1

0.75

User2

0.35

The optimal thresholds are different

Experimental Experimental ResultsResults Shape Regularization Results

Inner-Shape Regularization

The results of primitive shape classification and regularization

2.2.2. Composite Graphic Object Reco2.2.2. Composite Graphic Object Recognitiongnition

A “Partial” “Structural” Similarity Assessment Strategy is Proposed

B

O

A C

In order to suggest the user in an early stage, the system should recognize graphic object in an incomplete form.

2.2.2. Composite Graphic Object Reco2.2.2. Composite Graphic Object Recognitiongnition

A “Partial” “Structural” Similarity Assessment Strategy is Proposed

A1 A2 B1 B2

The similarity assessment strategy should not only invariant to shifting, rotation, mirroring, but also should invariant to inner distortion.

The Proposed The Proposed Approach Approach

The Source Object The Candidate Object

Spatial Relation Graph (SRGs)

The computational complicity is Pnm.

Conditioned Partial Permutation Algorithm

L1

L2 L3

L4

P1

P2 P3

P4

P5

Graphic Primitive Extraction

Line-segments, arc-segments,

and ellipses/circles

Performance Performance Evaluation Evaluation

Query Generating by adding noises and eliminating some parts.

Experimental results show that our approach can achieve good performance with noises for incomplete objects, and our approach is also invariant to shifting, rotation, mirroring, and inner distortions.

When the user draws 80% of his/her intended object (for users may miss some parts of the object inadvertently) with 10% distortion (this is similar to noises in real user drawing situation), R6 is nearly 90% (averagely of 304 graphic objects).

The Original Object Adding Noises Eliminating some partsThe Generated Query

2.3. Critical Technique 3: 2.3. Critical Technique 3:

User Adaptation User Adaptation

User adaptation is a classical problem in user interface study. Many pattern recognition problems are user specific, for users’ handwritings, drawing styles, and accents are different.

Rule-based feedback may yield “conflict” results due to its intrinsic deficiency, which may lose its general performance when it adapts to a specific user further.

Question: A triangle or a quadrangle?

An ambiguous case

SVM-incremental learning are introduced into the user adaptation problem of shape classification.

2.3.1. 2.3.1. QuestionsQuestionsFour questions need to be solved:

(1) Whether SVM-based Incremental learning can overcome “conflicts”?

(2) What is the advantage of Incremental leaning compared with repetitive learning?

(3) Which one is better, Syed’s or Xiao’s?

(4) Which structure is better, one-against-one and one-against-all?

2.3.2. Experiments2.3.2. ExperimentsExperimental Environments:

Feature Extraction (20-dimensional vector) by turning function

Virtual Sample Generation (with 40 samples each)

40 incremental training sets and two test sets are created

Training time, open-test precision, closed-test precision are tested for different algorithms and structures.

2.3.3. Answers2.3.3. Answers

(1) SVM-based incremental leaning can overcome “conflict”

(2) Incremental learning is much faster than repetitive learning without loss of precision

(3) Syed’s algorithm is better than Xiao’s

(4) One-against-one structure is much faster than one-against-all in our environments

Theoretical analysis and experimental results both show

PART III: THE PART III: THE SMART SMART SKETCHPAD SKETCHPAD SYSTEMSYSTEM

3.1. System architecture of Smart 3.1. System architecture of Smart SketchpadSketchpad

Stroke Recognizer

Closed-Shapes Open-Shapes

Shape Recognizer

Composite Graphic Object Recognizer

Graphic-DB

A New Stroke

Closed Shape Detector

Graphic Primitives Primitive Shapes

Visual Tokens

Interactive Editor

Feedback

Documents

Gesture

Semantic Level Application

Graphic-DB Manager

3.2. The Sketch-based User Interface of 3.2. The Sketch-based User Interface of Smart SketchpadSmart Sketchpad

The Sketching AreaThe Sketching Area

   

The just inputted shape are recognized and regularized

The just inputted shape are recognized and regularized

The candidate shape are shown

The candidate shape are shown

Part of the intended object are sketched

Part of the intended object are sketched

Candidate object name and their similarities are shown

Candidate object name and their similarities are shown

Candidate object list Candidate object list

Inputting two graphic objects and then delete them

Demo

3.3. UI Evaluation3.3. UI Evaluation

10 different subjects are required to draw the following two diagrams with traditional UI and the sketch-based UI. There are 304 objects listed in 26 stencils (with 12 each) for traditional UI. There are 6 objects can be shown in the Smart Toolbox for sketch-based UI.

Diagram 1 Diagram 2

A demo of inputting Diagram 2 by sketch

Drawing Time for Sketch 1 (s) Drawing Time for Sketch 2 (s) #User ID

Sketch-based Traditional Sketch-based Traditional 1 104 125 157 190 2 93 99 151 288 3 69 98 156 294 4 59 81 122 156 5 64 178 135 191 6 63 100 85 231 7 61 120 92 203 8 72 91 119 195 9 70 70 156 252

10 78 110 125 201

Drawing Time for Different Sketches Under Different UIs

Averagely, the sketch-based UI is 22.4% and 42.9% more efficient than the traditional toolbar button-based UI for sketch1 and sketch2, respectively.

The ultimate comments of all users unanimous agree that they’d like to choose the sketch-based UI instead of the traditional one.

PART IV: PART IV: SUMMAYSUMMAY

Algorithm Algorithm LevelLevel

Solution LevelSolution Level

System LevelSystem Level

(1) Agglomerate Point Filtering, Vertex Combination, Shape Fitting, Shape Regularization

(2) Conditioned Partial Permutation

(3) Comparison of SVM-based Incremental Learning Algorithms and Structures

(1) Interactive graphics inputting user scenario

(2) On-line graphics recognition: primitive shape classification and regularization; composite shape recognition.

(3) SVM-incremental learning for user adaptation in shape classification problem.

(1) A prototype system for conceptual/schematic designing tasks is implemented.

(2) User evaluation between the traditional and sketch-based UIs is performed.

Future WorksFuture Works

(1) How to perform stroke segmentation?

(2) How to cut down the computational cost and improve the recognition precision?

(3) How to make the system learn aggressively?

THANKSTHANKS