Preliminary Study on Vision-based Pen-and-Ink Drawing by a Robotic Manipulator

Intelligent Control Systems Laboratory

The Chinese University of Hong Kong

The Chinese University of Hong KongDept. of Mechanical and Automation

Engineering

Yan Lu, Josh Lam, Yeung Yam

Preliminary Study onVision-based Pen-and-Ink Drawing

by a Robotic Manipulator



MotivationBackgroundMethodologyDemonstrationSummary

Outline



Pen-and-Ink is a traditional form of art, which is still popular today.

Motivation



Skill-dependentSkill-dependent

Motivation

Time-consumingTime-consuming



A. Robot Platform 5-DOF manipulator 0.1mm movement

resolution Visual feedback by

CCD camera

Background



B. Pen-and-Ink Properties

1.Stroke is monochromatic;

2.Stroke contributes to both tone and texture;

3. Strokes work collectively, that is, no single one is of critical importance.

Background



Main challenge Only monochromatic strokes can be used to

convey three key aspects of an image: structural contents tone– brightness and darknessand textures

Methodology



A. Structural Contents

Step 1. Outline extraction– Canny edge detection

Step 2. Structural significance of edges– measured by the “lifetime” in the Gaussian scale space of the original image (Orzan et al. 2007)

http://artis.imag.fr/Publications/2007/OBBT07/

Step 3. Outline delineation by strokes of different widths according to structural significance

Methodology



Original image Structural significance



B. Tone Expression

The key is to place a stroke at a right location, based on the rule that strokes should be placed evenly: close together in dark areas, and widely spaced in bright areas.

To accomplish this hatching process, we need the vision system to provide visual feedback.

Methodology



B. Tone Expression (Cont’d)

Step 1. Capture the current drawing by camera, blur it by an average filter, transfer it to top view by homography, denoted as Iop;

Step 2. Compute the difference between current image and object image: Idf=Ioj-Iop

Methodology



B. Tone Expression (Cont’d)

Step 3. Determine the importance image as:

Step 4. Place the next stroke at the point where the value of the importance image is the largest

Step 5. Repeat 1~4 steps

Methodology

00

00

00

df

dfojdfidfi

ip I

IIaIII

Where i denotes the i -th stroke, and a is a parameter, empirically set as 0.1~0.5.



Object image Ioj Current image Iop Importance image Iip

Methodology

B. Tone Expression (Cont’d)Example of determining stroke position



Methodology

C. Texture Representation

Strokes should be appropriately orientated to represent desirable textures.

A natural way is to orientate a stroke according to the gradient direction of the original image. However, in practice, image gradient directions are often too noisy to be used directly.



C. Texture Representation (Cont’d)

Problem illustration

Methodology

Original image Local texture By gradient




Since strokes work collectively, a local texture should be expressed by a group strokes with consistent orientations.

We propose to select a set of “strong points” whose gradients are large enough to be believed, and calculate stroke direction by interpolation using a few nearest “strong points”.

Methodology




Stroke Orientation at p

Methodology

),(/)(

),(/)()()(

1

1

i

n

ii

i

n

iii

ppdpGrad

ppdpGradpp

Where pi is one of the n strong points nearest to p, θ(pi) is the gradient direction at pi , and d(p,pi) denotes the distance between p and pi .




Methodology

Strong points By gradient By gradient interpolation



Methodology

D. Stop Criterion

A criterion function is proposed to terminate the hatching process automatically as follows:

iop

i Ix

iop

Rx

i xIbxRiF 22 ))(())(()(

)()(0

)()()()()(

xIxI

xIxIxIxIxRwith i

opoj

iopoj

iopoji

The first component is a descending function wrt. stroke number i, while the second is an ascending one. They work together to determine the stop point.



Methodology

D. Stop Criterion (Cont’d)

The hatching process stops at the minimum point of the criterion function, which can be tuned by b.

0 2 4 6 8 10 12 14 16 18 208.06

8.08

8.1

8.12

8.14

8.16

8.18x 10

4

Stroke number i

Crit

erio

n fu

nctio

n F(

i)

×100 ×1000 2 4 6 8 10 12 14 16 18 20

7.88

7.9

7.92

7.94

7.96

7.98

8

8.02

8.04

8.06x 10

4

Stroke number i

Crite

rion

func

tion

F(i)

b=0.9, i=1400b=1.0, i=1000



Methodology

Input image

Outlines

Hatching

Terminate ?

++ _

Importance image

Visual feedback

Yes

Stroke position

No

Strong points

Stroke orientaion

Output image

Overview



Demonstration

i=0 i=300 i=700

i=1026 i=1283 i=1502



We have presented a robotic system for automated Pen-and-Ink rendering

Structural contents delineated according to structural significance of edges

Tone expression based on visual feedback Texture representation by interpolating local

gradient directions Automatic stop using a criterion function

Summary

Preliminary Study on Vision-based Pen-and-Ink Drawing by a Robotic Manipulator

Documents

Transcript of Preliminary Study on Vision-based Pen-and-Ink Drawing by a Robotic Manipulator