Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper...

13
The Visual Computer (2019) 35:1193–1205 https://doi.org/10.1007/s00371-019-01675-w ORIGINAL ARTICLE Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing Xinyue Zhang 1 · Yuanhao Li 1 · Zhiyi Zhang 1,2 · Kouichi Konno 3 · Shaojun Hu 1,2 Published online: 8 May 2019 © Springer-Verlag GmbH Germany, part of Springer Nature 2019 Abstract Chinese calligraphy is the artistic expression of character writing and is highly valued in East Asia. However, it is a challenge for non-expert users to write visually pleasing calligraphy with his or her own unique style. In this paper, we develop an intelligent system that beautifies Chinese handwriting characters and physically writes them in a certain calligraphy style using a robotic arm. First, we sketch the handwriting characters using a mouse or a touch pad. Then, we employ a convolutional neural network to identify each stroke from the skeletons, and the corresponding standard stroke is retrieved from a pre-built calligraphy stroke library for robotic arm writing. To output aesthetically beautiful calligraphy with the user’s style, we propose a global optimization approach to solve the minimization problem between the handwritten strokes and standard calligraphy strokes, in which a shape character vector is presented to describe the shape of standard strokes. Unlike existing systems that focus on the generation of digital calligraphy from handwritten characters, our system has the advantage of converting the user-input handwriting into physical calligraphy written by a robotic arm. We take the regular script (Kai) style as an example and perform a user study to evaluate the effectiveness of the system. The writing results show that our system can achieve visually pleasing calligraphy from various input handwriting while retaining the user’s style. Keywords Calligraphy · Beautification · Robotic arm · Optimization · Handwritten characters 1 Introduction Calligraphy, which is referred to as shufa in China and as shodo in Japan, is a traditional art that explores the beauty of human writing and cultivates a person’s character. However, it is very difficult for common people to write aesthetically Electronic supplementary material The online version of this article (https://doi.org/10.1007/s00371-019-01675-w) contains supplementary material, which is available to authorized users. B Shaojun Hu [email protected]; [email protected] Xinyue Zhang [email protected] Kouichi Konno [email protected] 1 College of Information Engineering, Northwest A&F University, Yangling, China 2 Key Laboratory of Agricultural Internet of Things, Ministry of Agriculture, Yangling, China 3 Faculty of Science and Engineering, Iwate University, Morioka, Japan elegant calligraphy without a large amount training because of the difficulty of mastering the complicated and subtle writ- ing skills needed to use ink and brush. A calligrapher may take decades to practice shufa (shodo) and establish his/her own writing style. Inspired by the fact that most people prefer to write char- acters using a hard-tipped pen rather than a soft-tipped brush, several systems have been proposed to convert a user’s hand- written characters into digital calligraphy with a specific style [12,13,26,28], and some one-click-output systems have been implemented to generate calligraphy from handwrit- ing efficiently [18]. Almost all these studies focused on the beautification process from an input sketch to a specific cal- ligraphy. However, the question of how to generate stylized calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and the interaction among the ink, brush, and paper, has an aesthetic olfactory and visual impact that cannot be obtained from the digital form of calligraphy. As robotics rapidly advances, human–robot interaction has been intro- duced for various tasks including stone stacking, garment folding, and cooking [4,7]. Thus, it is now possible to write 123

Transcript of Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper...

Page 1: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

The Visual Computer (2019) 35:1193–1205https://doi.org/10.1007/s00371-019-01675-w

ORIG INAL ART ICLE

Intelligent Chinese calligraphy beautification from handwrittencharacters for robotic writing

Xinyue Zhang1 · Yuanhao Li1 · Zhiyi Zhang1,2 · Kouichi Konno3 · Shaojun Hu1,2

Published online: 8 May 2019© Springer-Verlag GmbH Germany, part of Springer Nature 2019

AbstractChinese calligraphy is the artistic expression of character writing and is highly valued in East Asia. However, it is a challengefor non-expert users to write visually pleasing calligraphy with his or her own unique style. In this paper, we develop anintelligent system that beautifies Chinese handwriting characters and physically writes them in a certain calligraphy style usinga robotic arm. First, we sketch the handwriting characters using a mouse or a touch pad. Then, we employ a convolutionalneural network to identify each stroke from the skeletons, and the corresponding standard stroke is retrieved from a pre-builtcalligraphy stroke library for robotic armwriting. To output aesthetically beautiful calligraphywith the user’s style, we proposea global optimization approach to solve the minimization problem between the handwritten strokes and standard calligraphystrokes, in which a shape character vector is presented to describe the shape of standard strokes. Unlike existing systems thatfocus on the generation of digital calligraphy from handwritten characters, our system has the advantage of converting theuser-input handwriting into physical calligraphy written by a robotic arm. We take the regular script (Kai) style as an exampleand perform a user study to evaluate the effectiveness of the system. The writing results show that our system can achievevisually pleasing calligraphy from various input handwriting while retaining the user’s style.

Keywords Calligraphy · Beautification · Robotic arm · Optimization · Handwritten characters

1 Introduction

Calligraphy, which is referred to as shufa in China and asshodo in Japan, is a traditional art that explores the beauty ofhuman writing and cultivates a person’s character. However,it is very difficult for common people to write aesthetically

Electronic supplementary material The online version of this article(https://doi.org/10.1007/s00371-019-01675-w) containssupplementary material, which is available to authorized users.

B Shaojun [email protected]; [email protected]

Xinyue [email protected]

Kouichi [email protected]

1 College of Information Engineering, Northwest A&FUniversity, Yangling, China

2 Key Laboratory of Agricultural Internet of Things, Ministryof Agriculture, Yangling, China

3 Faculty of Science and Engineering, Iwate University,Morioka, Japan

elegant calligraphy without a large amount training becauseof the difficulty ofmastering the complicated and subtle writ-ing skills needed to use ink and brush. A calligrapher maytake decades to practice shufa (shodo) and establish his/herown writing style.

Inspired by the fact that most people prefer to write char-acters using a hard-tipped pen rather than a soft-tipped brush,several systems have been proposed to convert a user’s hand-written characters into digital calligraphy with a specificstyle [12,13,26,28], and some one-click-output systems havebeen implemented to generate calligraphy from handwrit-ing efficiently [18]. Almost all these studies focused on thebeautification process from an input sketch to a specific cal-ligraphy. However, the question of how to generate stylizedcalligraphy and physically write it on paper is interestingbecause the physical form of calligraphy, with the scent ofink and the interaction among the ink, brush, and paper,has an aesthetic olfactory and visual impact that cannot beobtained from the digital form of calligraphy. As roboticsrapidly advances, human–robot interaction has been intro-duced for various tasks including stone stacking, garmentfolding, and cooking [4,7]. Thus, it is now possible to write

123

Page 2: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

1194 X. Zhang et al.

Fig. 1 Our system canintelligently write visuallypleasing Chinese calligraphygiven input handwriting. (Left)The robotic arm writing aChinese character “Wei” in Kaistyle; (middle) inputhandwritten characters “WeiLai”, which means “future” inEnglish; (right) Kai-stylecalligraphy output afterbeautification and writing onpaper

simple strokes [16] and characters using robots [1,6,10,14].Although Ma and Su [15] proposed a standard of aestheticsevaluation for robotic writing of Chinese calligraphy, it ischallenging for the current writing results of robotic arms tomeet the aesthetic requirements of calligraphy with a partic-ular user’s style because it needs to consider the complicatedstructure and transition of strokes when using ink and brush.

In this paper, we propose an intelligent Chinese calligra-phybeautificationmethod for robotic arms andbuild a systemto generate calligraphy with a user’s writing style from theinput handwritten characters. Figure 1 shows the intelligentwriting results from Chinese handwriting to regular script(Kai) calligraphy. First, the user handwrites characters inskeleton form using a mouse or a touch pad. Based on anestablished calligraphy stroke library, a global optimizationmethod is employed to generate the optimal stroke. Finally,the optimized stroke is written physically on a piece of paperusing ink and a brush by the robotic arm.

This paper makes the following contributions regardingthe writing of aesthetically pleasing Chinese calligraphyfrom handwriting:

– a global optimization approach to generate the optimalstroke for a robotic arm while retaining both the callig-raphy and handwriting characteristics, in which a shapecharacter vector (SCV) is proposed to describe the shapeof a stroke skeleton;

– an intelligent system to physicallywrite calligraphyusingrobotic arms.

2 Related work

Generally, the process of converting handwritten charactersto calligraphy consists of four main parts: character writing

and stroke extraction, stroke recognition, stroke beautifica-tion, and robot control. With regard to character writing andstroke extraction, previous studies can be divided into twocategories. One is stroke extraction based on writing images,and the other is stroke extraction from a user interface. Sunet al. [20] extracted strokes in terms of geometric propertiesaccording to character contours. Chen et al. [2] improvedthe stroke extraction algorithm based on Delaunay triangu-lation to effectively extract strokes from images. Zhang et al.[28] implemented a user interface tomanually extract controlpoints from the images. Kim et al. [9] presented a seman-tic vectorization approach to segment each stroke from theimage of a character. The other approach is the direct record-ing of the stroke positions and sequences fromauser interfacewhile a user is writing [27]. The second approach is moreintuitive because the stroke information is determined whilethe user is writing a character.

Once the individual stroke is extracted, the second step isstroke recognition. Zhang et al. [28] utilized feature cornerpoints and a directed graph to identify the type of each stroke.Sun et al. [19] adopt the Mahalanobis distance for similaritymeasurement to classify types and styles of different Chinesecalligraphy.Recently, convolutional neural networks (CNNs)have provided an efficient solution for handwriting recogni-tion.Gupta et al. [5] proposed an offline recognition approachfor recognizing handwritten English words using neural net-works. Syamlan et al. [22] used two neural networks with19 and 22 hidden layers to identify capital letters with ahigh recognition rate. Xiao et al. [24] designed a nine-layerCNN for handwritten Chinese character recognition consist-ing of 3755 classes, and their highest recognition rate reached97.3%. Thus, the CNN-based recognition approaches showgreat potential in the area of handwriting recognition.

After obtaining the type of each stroke, we can beautifyeach stroke and convert it to calligraphy with a specific style.

123

Page 3: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing 1195

(d) Extract parameters

lij

wij

θij

(e) Calligraphy database

(f) Corresponding stroke

(h) Write by robotic arm(g) Optimized character

(a) Chinese handwriting (b) Extract strokes

137

conv

pooling

conv

pooling

...

Full-connet

(c) Recognize with CNN

Fig. 2 Overview of our robotic writing system

Zhang et al. [28] achieved the beautification from handwrit-ing to calligraphy by controlling the contour point of eachstroke. Although this approach could generate elegant callig-raphy, the restrictions on input handwritten characters are toostrict for non-expert users: It is difficult for them to write neatcharacters and adjust the corresponding parameters. Liu et al.[13] divided a handwritten stroke into segments and beauti-fied the contours of Kai-style calligraphy using cubic Beziercurves. Li et al. [12] presented an interpolation approach tobeautify Chinese handwriting, and Yi et al. [26] proposed adigital ink generatingmethod for offlineChinese handwritingcharacters. Although these methods have ability to generatevisually plausible digital calligraphy, they are unsuitable forcreating physical calligraphy for robotic arm writing. More-over, some of these beautification methods require the neatinput of handwritten characters, so they are not suitable fornon-expert users.

Hashiguchi et al. [6] utilized variable impedance controlto implement a human–robot cooperative calligraphy task.Lo et al. [14] developed a facility to capture and analyse thebrush footprint of Chinese calligraphy. Chao et al. [1] cap-tured human body gestures through a Kinect and then used arobotic arm to control a hard-tip pen to write characters on awhiteboard. All these methods focused more on the dynamicand kinematic controls of robotic arms, but few of them haveconsidered the deformation of a soft-tipped brush on a paperor the generation of stylized calligraphy. Yao et al. [25] haveconstructed a Chinese calligraphy database to generate thetrajectory of the writing brush for a robot arm. However, theinput image patterns of Chinese calligraphy should be neatfor robotic writing. Thus, it is not feasible for non-expertusers. Muller et al. [16] introduced a novel learning-basedapproach to write single strokes of calligraphy charactersusing a robotic arm, while the approach concentrated onwriting single strokes rather than complete characters. In

addition, they have not considered the users’ writing style.Sun et al. [21] presented a local weighted linear regressionapproach to implement the mapping from stroke representa-tion to brush trajectory, but it required human demonstrationsand it is difficult to generate stylized calligraphy with theusers’ input.

3 Overview

In this paper, we propose an intelligent writing system toconvert handwritten characters to aesthetically pleasing cal-ligraphy for robotic writing. Figure 2 shows the workflow ofour writing system, which consists of four parts:

First, users write a complete Chinese character in a skele-ton form using a mouse or touch pad, as shown in Fig. 2a.Then, the systemautomatically extracts the individual strokesand obtains the coordinates of each stroke in order, as shownin Fig. 2b. Next, a CNN-based recognition approach identi-fies the type of stroke, as shown in Fig. 2c. We introduce theinput and recognition of handwriting in Sect. 4.

To generate calligraphy strokes in Kai style for roboticwriting, several parameters are extracted from standard cal-ligraphy stroke images, and the database of stroke parametersis built as shown in Fig. 2d–e. According to the recognizedtype of handwritten stroke from Fig. 2c, the same kind ofcalligraphy stroke in the database is retrieved, as shown inFig. 2f. We describe this part in Sect. 5.

Once the two types of strokes are obtained, a global opti-mization method is employed to generate the most suitablestroke for robotic writing, as shown in Fig. 2g. On one hand,the target stroke should be close to the standard calligraphyin the database. On the other hand, it also should preserve theuser’s handwriting characteristics. Finally, we input the opti-mized control parameters of the strokes to a robotic arm and

123

Page 4: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

1196 X. Zhang et al.

Fig. 3 Handwritten character“Cai”

Input48×48

C1 6×46×46

S2 6×23×23

C3 12×22×22

S4 12×11×11

Convolution Subsampling

C512×10×10

Convolution Subsampling Fully-connect3×3

Convolution Subsampling

S6 12×5×5

20 strokes

Output F7

2×2 2×2 2×2 2×2 2×2

Heng

Shu

Pie Na

Fig. 4 Architecture of the CNN in our system

write the calligraphy on a piece of paper, as shown in Fig. 2h.We introduce these two parts in Sect. 6. Finally, we evaluatethe effectiveness and usability of the system in Sect. 7.

4 Handwriting input and recognition

Handwriting input and recognition is the basis for our roboticwriting system. To conduct these steps easily and intelli-gently, we developed an intuitive user interface to record auser’s handwritten characters and extract individual strokes.Then, a CNN-based recognition approach was employed toidentify the type of each stroke.

4.1 Character writing and stroke extraction

Many approaches have been presented to acquire strokesfrom a completed character [2,20,28]. For convenience, weextract stroke coordinates directly from handwriting usingan interface rather than from images, because it is easier torecord the stroke coordinates and sequences.

Figure 3 shows a user writing the Chinese character “Cai”with a touch pad. The character is displayed in an interface.To generate a concise skeleton for roboticwriting, every inputstroke is re-sampled adaptively to form a polyline [8]. Blacklines in the image refer to the handwritten strokes “Heng”,“Shugou”, and “Pie” in skeleton form. Red dots indicate thevectorization skeleton points extracted in real time. After

the handwriting is finished, the system automatically savesthese stroke points in sequential order in a file. Furthermore,the obtained stroke points are used to generate the strokeimage for subsequent CNN recognition. The main advantageof the user interface is that the generated stroke informationis suitable for calculating coordinates and driving the jointmotion of a robot arm to form a writing trajectory.

4.2 Stroke recognition with CNN

Recent studies have suggested that CNNs have a high recog-nition rate for handwritten characters [5,22,24]. Thus, weuse a CNN to recognize the extracted strokes. To avoid lowrecognition rate caused by image quality, the stroke imagesare preprocessed into a specified size. The previous studiesshowed that the rate of recognition is increased as the imagesize increases. However, the number of network layers andforward decoding time also increase. In our system, we bal-ance the image size and number of layers. The image size inan input layer is 48 × 48, and the number of layers is eight.In addition, because there is no public handwritten strokedataset available for training, we had to build a training set.Thus, we collected 20 kinds of strokes. Ten samples of eachtype of stroke were written by 20 different people, giving atotal of 200 training images for each stroke. Meanwhile, atenfold-cross verification is introduced to avoid overfitting.The architecture of the CNN used in our system is shown inFig. 4.

123

Page 5: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing 1197

Fig. 5 Parametric model ofstroke “Heng” and its variousresults written by the roboticarm with a brush. a Parametricmodel of a standard Kai-style“Heng”. b Deformation of abrush during writing. c “Heng”strokes of various widthscorresponding to the variouschanges in height of a brushcontrolled by a robotic arm

After 100 epochs of iterative training, the recognition rateon the validation set converged to 97%. Then, we collected20 kinds of strokes from five different people to form threetest datasets. The result shows that the average recognitionrate of the test datasets reaches 93%.

5 Stroke database establishment

To generate elegant calligraphy strokes for robotic writing,a unique parameter model was built for each type of stroke.In our system, 20 kinds of Kai-style strokes, which wereobtained from Tian’s book The Elaboration of Ancient andModern Famous Kai Calligraphy [23], are used as the pro-totypes for creating a parametric model.

5.1 Stroke parameterization

A standard stroke consists of rich contour and skeleton infor-mation. However, it is difficult to draw the precise strokecontour because a brush is soft and difficult to control. Thus,we take the skeleton of a standard stroke into consideration.First, the skeletons from the 20 typical stroke images wereextracted using a standard thresholding and thinning algo-rithm [17]. To establish the parametricmodel for the standardKai strokes, a sequence of triplets, (l ji , θ

ji ,w

ji ), is introduced

to represent a stroke from the skeleton shown in Fig. 5 (a),where subscript i denotes the i th stroke and superscript jrefers to the j th segment of the i th stroke. Finally, l, θ , andw

represent the feature length, corner angle, and average widthof the stroke, respectively. Each stroke can be separated intoseveral small segments according to the start point, cornerpoints, and end point, as shown in Fig. 5a. In this figure, thefirst segment is from the start point to the first corner point,and the last one is from the end corner point to the end point.The middle segment is linked by the two adjacent corners.Similar to the work of Zhang et al. [28], we define the shapeof the head and the tail of the stroke as rigid segments (the redarea in Fig. 5a), and the transition segment between the rigidsegments as a flexible segment (the green area in Fig. 5a).Note that the rigid segment contains the important features ofKai-style calligraphy, and we only apply a rigid transforma-tion to it. On the contrary, the flexible segment can be resizedflexibly according to the user’s writing style, and thus, we canuse a non-rigid transformation (such as a non-uniform scaletransformation) on it. Hence, we represent each segment bya triplet.

As for parameter w, we approximate the relationshipbetween an average stroke width and the descent of therobotic arm brush. Suppose horig is the length of a brushtip and hrest is the vertical length of the brush tip after defor-mation, as shown in Fig. 5b. Then, the descent of the brushh, which can be controlled easily by the robotic arm, is rep-resented by horig − hrest. Intuitively, a large descent of thebrush will cause a large bend in the brush, and eventuallywrite a wider stroke on a paper. To determine the relation-ship between w and h, we designed an experiment to collectthe actual stroke width corresponding to every 1 mm descent

123

Page 6: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

1198 X. Zhang et al.

Fig. 6 Comparison of standard strokes with strokes physically writtenby a robotic arm. a Original standard strokes; b robot-written strokesusing the parametric model extracted from a

Fig. 7 Diagram of a original handwriting, b skeleton of standard cal-ligraphy, c skeleton of target stroke

of the brush tip. Figure 5c shows the variouswriting results of“Heng” with different widths. Next, we measured the widthsof the calligraphy strokes at three positions by cutting off thehead and tail of the stroke. Then, the average value of the threewidths is assigned tow. The brush tipwas lowered from1mmto7mm, and the corresponding strokewidthswere 2, 3.2, 3.8,4.5, 5.5, 6.2, and 6.8 mm. Finally, a standard least-squaresfitting approach was employed to approximate the relation-ship between w and h. We obtained w = 0.7893h + 1.414,for which the summed square of residuals is 0.1311, and thecoefficient of determination R2 is 0.9925, which indicatesthat the prediction fits the data well in our system.

5.2 Construction of stroke database

The parametric model for the 20 standard strokes we built isable to direct the motion of a robotic arm by controlling thethree parameters to ensure that the brush follows the pre-

determined trajectory. Figure 6 shows Tian’s 20 standardcalligraphy strokes [23] and the corresponding calligraphystrokeswritten by the robotic arm.Each stroke in the databasehas a different parametric model that plays an important rolein the control of the robotic arm. However, the results ofwriting the same stroke with the same parameters are highlyvariable because the initial state of a soft brush has a substan-tial impact on the writing results. Therefore, we manuallyadjust the brush before writing a new stroke if there is anirreversible deformation in the brush tip.

6 Calligraphy beautification and writing

Calligraphy beautification is the process of converting hand-written characters to calligraphywith a specific style. Severalapproaches have been proposed to achieve good beautifica-tion results by controlling the contour points of strokes usingBezier curves [13,28]. However, they have three limitations:(1) the input of the handwritten characters must be neat andbeautiful, so it is not suitable for non-expert users. (2) Itis difficult to preserve both the user’s writing style and thestandard calligraphy style. (3) The simulated calligraphy isinappropriate for robotic writing. In this paper, we introducea global optimization approach to generate Chinese calligra-phy by preserving both the handwriting characteristics andcalligraphy style. For every stroke, an optimization functionis employed to generate a target stroke for robot calligraphy.

6.1 Global optimization of the target strokes

Unlike simulated calligraphy, it is difficult to precisely con-trol and write calligraphy using a robotic arm and a brush.Thus, we control the skeleton points [25] rather than the con-tour points of each stroke. Because we have already obtainedthe skeleton points of the handwritten stroke and the standardcalligraphy stroke, our goal is to find the optimal skeletonof the target strokes. To achieve this, we introduce a posi-tion similarity F(w, υ) and an SCV G(u, w) to measure thedifference between the input skeleton points and the targetskeleton points, and define a smoothnessmeasurement H(w)

to constrain the deformation of the target skeleton points.To preserve the position of the original handwriting, the

generated skeleton should be close to the user’s handwrittenskeleton. This leads us tominimize aweightedEuclidean dis-tance between the generated and target points. The resultingposition similarity function F(w, υ) is calculated as follows:

F(w, υ) =∑

i

‖wi − υi‖ =∑

i

(wi − υi )T(wi − υi ) (1)

where wi represents the coordinates of the control pointsafter optimization and υi represents the original handwrittenskeleton points, as shown in Fig. 7a, c.

123

Page 7: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing 1199

Fig. 8 Optimization process of the character “Cai”. a Original hand-written characters; b handwritten characters in a 90 × 90 grid; coptimized character; d–g results of different combinations of parame-

ters α, β, and γ for the stroke “Pie”. Red lines represent the handwrittenskeletons, blue lines denote the skeletons in standard calligraphy, andblack lines refer to the optimized result

The second goal is to preserve the shape of the correspond-ing strokes in the standard calligraphy database. To achievethis goal,wedefine anSCVfrom three adjacent control pointsto represent the shape of a stroke. Let ui , ui+1, and ui+2 beadjacent points in the standard skeleton, and wi , wi+1, andwi+2 be adjacent points in the generated skeleton, as shown

in Fig. 7b, c. The SCVs of the two skeleton points are denotedby pi = ui+1 + ui+2 − 2ui and qi = wi+1 + wi+2 − 2wi .To preserve the features of standard calligraphy strokes, theobjective function G(u, w) of the SCVs can be calculated by

G(u, w) =∑

i

‖pi − qi‖ =∑

i

fsqrt(pi − qi ) (2)

123

Page 8: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

1200 X. Zhang et al.

In addition, the smoothness of the target stroke is alsoa key factor that affects the appearance of calligraphy. Letwi , wi+1, and wi+2 be adjacent points in the generatedstroke, and the normalized direction vector of wi and wi+1

is ni = wi+1−wi‖wi+1−wi‖ . Thus, the constraint function H(w) for

the generated stroke is

H(w) = ni+1 − ni‖ni+1 − ni‖ =

i

fsqrt(ni+1 − ni ) (3)

Finally, the probable points of the generated strokeare obtained by solving the minimization of F(w, υ) +G(u, w) + H(w) as follows:

X̂ = argminx

[∑

i

(wi − υi )T(wi − υi )

]

+ β

[∑

i

fsqrt(pi − qi )

]

+ γ

[∑

i

fsqrt(ni+1 − ni )

]]

(4)

where α, β, and γ are weight coefficients. Then, a conjugategradient algorithm is used to solve Eq. 4 [11].

To obtain optimal stroke writing results, we constantlychange the three weight parameters α, β, and γ , and choosea good combination of parameters for robotic writing. Fig-ure 8a, b shows the original handwritten Chinese character“Cai” and the extracted strokes “Heng”, “Shugou”, and “Pie”in a specified grid. To observe the difference in the optimizedstrokes due to different combinations of parameters, we setthe range of each parameter to [0.1, 1] and changed one ofthe three parameters while keeping the remaining parametersconstant.

Figure 8d–g shows the optimization results of the stroke“Pie”with four combinations ofα,β, and γ . Just as expected,large γ (Fig. 8e) generates a smoother skeleton (black line)than small γ (Fig. 8d) when α and β are fixed. To see theeffect of β, we fix α and γ , and the results show that largeβ preserves the shape of the skeleton in standard calligra-phy (blue line) better than small β, as shown in Fig. 8f, g.Finally, we find that large α preserves the positions of theoriginal handwritten characters (red line) better than smallα, as shown in Fig. 8e–g when β and γ are fixed. The phe-nomenon becomes more obvious if we set α to a large valueand β as a very small value (see Fig. 8f). Based on this anal-ysis, we find that when the combination is α = 0.5, β = 1,and γ = 1, the optimized skeleton of the target stroke notonly maintains the unique characteristics of calligraphy, butis also close to the handwritten characters. Thus, we selectedthis combination of parameters for the optimization of stroke“Pie”. For different strokes such as “Heng” and “Shugou”,

we just need to fine tune α and β based on this combination.The optimized character is shown in Fig. 8c.

6.2 Control of robotic armwriting

The final writing task is implemented using a Dobot roboticarm [3]. All motions of the Dobot robotic arm are realized byfour-degree-of-freedom joints. After optimization, the coor-dinates of the three-dimensional control points (x , y, z) ofeach strokemust be accurately transmitted to the robotic arm.The (x , y) coordinates are determined by the optimized two-dimensional control points, and the height values z of thepoints are derived from the standard stroke, as described inSect. 5. However, the position of the newly generated pointsis changed after optimization. Thus, it is inappropriate todirectly use the z value from the standard stroke database.

To solve this problem, we first expand the points of anoptimized stroke and retrieve the corresponding points fromthe standard stroke database. The process of locating the cor-responding points is described by Algorithm 1.

Algorithm 1 locateCorrespondingPoints(u[],w[])// u[] represent the skeleton points from standard calligraphyand w[] are the optimized skeleton points1: for two adjacent points of w[] do2: d[] = |wi+1 − wi | // Obtain distance;3: end for4: newPts[] = ∅;5: newPts[] = newPts[] ∪ {u0} // Save the initial points;6: for each d j in d[] do7: Select the first uk once |uk − u0| ≥

j∑s=0

ds ;

8: newPts[] = newPts[] ∪ {uk};9: end for10: Return the corresponding points newPts[];

Next, the z value of the control points are linearly inter-polated between the extracted height values of the adjacentcorresponding points from the standard database. Finally,these points are transmitted to the robotic arm through theDobotStudio interface and thefinalwriting task is performed,as illustrated in Fig. 9.

7 Results and discussion

Our robotic writing system was implemented in a PC con-figured with a 2.8 GHz CPU, 8 GB RAM, and Windows10 operating system. The handwritten user interface wasdeveloped in C++ with a Wacom CTH-690 touch pad. Thecalligraphy beautification was tested in MATLAB, and thecalligraphy writing was performed by a Dobot Magicianrobotic arm, as shown in Fig. 1. To evaluate the effective-

123

Page 9: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing 1201

Fig. 9 Screenshot of the Dobot Studio interface for robotic writing

ness of our system, we compared the calligraphy writingresults before and after beautification. Then, we collecteddifferent handwritten characters and tested the correspond-ing results of the calligraphywriting. In addition, a user studywas conducted to evaluate the usability and effectiveness ofour system.

7.1 Comparison with non-optimized calligraphy

As mentioned in Sect. 5.1, each stroke can be segmentedinto rigid parts and flexible parts. A question worth con-sidering is whether the rigid part of each stroke should be

preserved for the optimized strokes. For this issue, a com-parative experiment was conducted on the Chinese character“Cai”, as shown in Fig. 10.

First, we directly utilized the two-dimensional points fromhandwritten characterswith a constant descent height h of thebrush for writing, as shown in Fig. 10a. Because the descentheight did not change for a certain writing, the generatednon-optimized calligraphy did not have the features of Kai-style calligraphy at the head and tail parts (see the standardcalligraphy strokes in Fig. 6) and thus lacked the aestheticsof calligraphy. Moreover, we observed that a small descenth preserved the trajectory of handwritten characters better

123

Page 10: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

1202 X. Zhang et al.

Fig. 10 Comparison results of writing calligraphy “Cai”. From left to right: a no optimization results as the height of the brush is lowered; b resultsof calligraphy writing after optimization without rigid segment replacement; c results of calligraphy writing after optimization with rigid segmentreplacement

than a large descent, because the large descent induced largebending of the brush that caused the rigid segments to beincorrectly written.

Next, we tested the writing result of the same character“Cai” with three optimized strokes, as shown in Fig. 10b. Wetreated all the segments of the stroke as flexible segments. Incomparisonwith non-optimizedwriting results, the results ofcalligraphy writing after optimization showed some featuresofKai-style calligraphy around the corner parts of the strokes.However, some rigid segments were still not fully reflected,as indicated by the blue circles of Fig. 10b.

Given the results of the second experiment, we took therigid segments into consideration in addition to the optimiza-tion of the flexible segments. Specifically, each stroke wassegmented into rigid parts and flexible parts and handled sep-arately. Forflexible segments,weused the optimized skeletonto control the robotic writing. However, the lengths of rigidsegments were kept unchanged, and we directly retrieved thedescent height information from the pre-built standard Kai-style calligraphy. Figure 10c shows the results of calligraphywriting after optimization with rigid segment replacement.In contrast to the results of the second experiment, we foundthat all the features of the rigid segments were preservedin this experiment (see the emphasized head and tail partsindicated by the red circles of Fig. 10c). Therefore, robotic

Fig. 11 Handwritten Chinese characters from a non-expert users, b aprofessional user, and the corresponding Kai-style calligraphy writtenby the robotic arm

writing with optimized strokes and rigid segment replace-ment not only maintains the characteristics of handwrittencharacters better, but also preserves the Kai style of calligra-phy. As a result, aesthetically pleasing Chinese calligraphywith a user’s unique style can be created.

123

Page 11: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing 1203

Table 1 User study questionnaire

Questions Choices

Q(1) The beautification method is useful Very Somewhat Neutral Rarely Not at all

Q(2) Robotic writing preserves the style of handwriting Very Somewhat Neutral Rarely Not at all

Q(3) Robotic writing preserves the features of Kai-style calligraphy Very Somewhat Neutral Rarely Not at all

Q(4) Robotic writing is aesthetically pleasing Very Somewhat Neutral Rarely Not at all

Q(5) The overall process is time consuming Very Somewhat Neutral Rarely Not at all

Q(6) I would recommend the system to calligraphy amateurs Very Somewhat Neutral Rarely Not at all

Fig. 12 Statistics of the questionnaire feedback from 15 participants

7.2 Results of different users’ input

To show the flexibility of our method, we collected sixhandwritten Chinese characters and wrote the correspond-ing Kai-style calligraphy using the robotic arm, as shown inFig. 11. Four of them, “Cai”, “Hua”, “Li”, and “Shi”, werewritten by common users, and two of them, “Wei” and “Lai”,were written by a professional user. The results demonstratethat our calligraphy beautification approach for robotic writ-ing preserves both the styles of user’s handwriting and thestyle of Kai calligraphy. Especially in Fig. 11a, the roboticarmwas able to write visually pleasing Kai-style calligraphy,even though the user-input handwritten characters were notneat.

7.3 User study

To evaluate the robotic writing system, a user questionnairewas employed, as shown in Table 1. Fifteen users partic-ipated in the study. Three of them were expert users whocould write beautiful handwritten characters, and 12 of themwere amateurs. The results of their responses to the ques-tionnaire are shown in Fig. 12. We observed that 87% of theusers agreedwith the usability of our system, and 67%agreedthat calligraphy written by robotic arm preserved the style ofhandwriting well, whereas only 6.5% of the users had neutralopinions. Moreover, 87% of the users considered the outputcalligraphy to have the characteristics of Kai style, and 80%of them admitted the results were aesthetically pleasing. Inaddition, although 67% of the users were willing to recom-

mend the system to calligraphy amateurs, 74% of the usersraised the shortcoming of the system’s slow speed. Therefore,our robotic calligraphy writing system is close to achievingthe expected goal of generating visually pleasing calligraphywhile preserving the user’s writing style.

7.4 Limitations

Although we have achieved visually pleasing calligraphywriting using a robotic arm, our system still has three lim-itations: One limitation is the speed, which occurs becausewe have to manually tune the initial parameters for generat-ing optimized strokes. Second, our current database containsonly 20 strokes and does not cover some complex strokes.Thus, it is difficult for the system to write complex Chinesecharacters. The third limitation is the influence of the physi-cal form of a brush. Different brushes have different materialproperties, and the same brush loadedwith a different amountof ink will physically write different strokes, even when thesame parameters are used.

8 Conclusions and future work

We proposed a global optimization approach to generatevisually pleasingChinese calligraphy fromhandwritten char-acters for robotic writing. In addition, an intelligent systemwas built that consists of a simple interface for recording auser’s handwriting and extracting individual strokes from atouch pad, and a robotic arm for physically writing the opti-mized calligraphy on paper. To achieve a high recognitionrate, a CNN-based approach was utilized to identify the typeof each stroke. Unlike the traditional approach to beautifycalligraphy from the contours of strokes, we mainly consid-ered the skeleton of each stroke because a real brush is soft,and it is therefore difficult to draw detailed contour informa-tion. To preserve both the user’s writing style and the styleof the calligraphy, we segmented each stroke into rigid partsand flexible parts. Then, we applied the style of standard Kai-style calligraphy to the rigid parts and applied the optimizedstrokes with the user’s writing style to the flexible parts. The

123

Page 12: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

1204 X. Zhang et al.

results of several experiments and a user study show that ourrobotic writing system not only maintains the user’s uniquewriting style, but also preserves the Kai style of calligra-phy. In future, we intend to improve the stroke database andimplement efficient calligraphy writing for complex Chinesecharacters. Moreover, we plan to design a control systemto automatically detect and adjust the physical state of thebrush.

Acknowledgements Wethank theCGI2019 reviewers for their thought-ful comments. The work is supported by the NSFC (61303124), NSBRPlan of Shaanxi (2019JM370) and the Fundamental Research Funds forthe Central Universities (2452017343).

Compliance with ethical standards

Conflict of interest All authors declare that they have no conflict ofinterest.

References

1. Chao, F., Lv, J., Zhou, D., Yang, L., Lin, C., Shang, C., Zhou,C.: Generative adversarial nets in robotic Chinese calligraphy. In:2018 IEEE International Conference on Robotics and Automation(ICRA), pp. 1104–1110 (2018)

2. Chen, X., Lian, Z., Tang, Y., Xiao, J.: A benchmark for strokeextraction of Chinese character. Acta Sci. Nat. Univ. Pekin. 52(1),49–57 (2016)

3. Dobot Robotic Arms: https://www.dobot.cc/. Accessed on Feb 10,2019

4. Furrer, F., Wermelinger, M., Yoshida, H., Gramazio, F., Kohler,M., Siegwart, R., Hutter, M.: Autonomous robotic stone stackingwith online next best object target pose planning. In: 2017 IEEEInternational Conference on Robotics andAutomation (ICRA), pp.2350–2356 (2017)

5. Gupta, A., Srivastava, M., Mahanta, C.: Offline handwritten char-acter recognition using neural network. In: 2011 IEEE InternationalConference on Computer Applications and Industrial Electronics(ICCAIE), pp. 102–107 (2011)

6. Hashiguchi, H., Arimoto, S., Ozawa, R.: Control of a hand-writing robot with DOF-redundancy based on feedback in task-coordinates. J. Robot. Mechatron. 16, 381–387 (2004)

7. Igarashi, T., Inami, M.: Exploration of alternative interaction tech-niques for robotic systems. IEEE Comput. Graph. Appl. 35(3),33–41 (2015)

8. Igarashi, T.,Matsuoka, S., Tanaka,H.: Teddy:A sketching interfacefor 3D freeform design. In: SIGGRAPH’99, pp. 409–416, NewYork (1999)

9. Kim, B., Wang, O., Öztireli, A.C., Gross, M.: Semantic segmenta-tion for line drawing vectorization using neural networks. Comput.Graph. Forum 37(2), 329–338 (2018)

10. Kim, S., Jo, J., Oh, Y., Oh, S., Srinivasa, S., Likhachev, M.:Robotic handwriting: multi-contact manipulation based on reac-tional internal contact hypothesis. In: 2014 IEEE/RSJ InternationalConference on Intelligent Robots and Systems, pp. 877–884 (2014)

11. Krizek, M., Neittaanmki, P., Glowinski, R., Korotov, S.: ConjugateGradient Algorithms and Finite ElementMethods. Springer, Berlin(2004)

12. Li, H., Liu, P., Xu, S., Lin, S.: Calligraphy beautification methodfor Chinese handwritings. In: 2012 4th International Conferenceon Digital Home, pp. 122–127 (2012)

13. Liu, L., Xia, W., Jin, L., Mao, H., Tian, F.: A Kai style contourbeautificationmethod for Chinese handwriting characters. In: 2010IEEE International Conference on Systems, Man and Cybernetics,pp. 3644–3649 (2010)

14. Lo, K.W., Kwok, K.W., Wong, S.M., Yam, Y.: Brush footprintacquisition and preliminary analysis for Chinese calligraphy usinga robot drawing platform. In: 2006 IEEE/RSJ International Con-ference on Intelligent Robots and Systems, pp. 5183–5188 (2006)

15. Ma, Z., Su, J.: Aesthetics evaluation for robotic Chinese calligra-phy. IEEE Trans. Cogn. Dev. Syst. 9(1), 80–90 (2017)

16. Mueller, S., Huebel, N., Waibel, M., D’Andrea, R.: Roboticcalligraphy-learning how to write single strokes of Chinese andJapanese characters. In: 2013 IEEE/RSJ International Conferenceon Intelligent Robots and Systems, pp. 1734–1739 (2013)

17. Okutomi, M.: Digital Image Processing. The Computer GraphicArts Society (CG-ARTS), Tokyo (2015)

18. Online Chinese Calligraphy Generator: https://www.zhenhaotv.com/. Accessed on Dec. 4, 2018

19. Sun, Y., Ding, N., Qian, H., Xu, Y.: A robot for classifying Chinesecalligraphic types and styles. In: 2013 IEEE International Confer-ence on Robotics and Automation, pp. 4279–4284 (2013)

20. Sun, Y., Qian, H., Xu, Y.: A geometric approach to stroke extrac-tion for the Chinese calligraphy robot. In: 2014 IEEE InternationalConference on Robotics and Automation (ICRA), pp. 3207–3212(2014)

21. Sun, Y., Qian, H., Xu, Y.: Robot learns Chinese calligraphy fromdemonstrations. In: 2014 IEEE/RSJ International Conference onIntelligent Robots and Systems, pp. 4408–4413. IEEE, New York(2014)

22. Syamlan, A.T., Nurhadi, H., Pramujati, B.: Character recognitionfor writing robot control using ANFIS. In: 2015 International Con-ference on Advanced Mechatronics, Intelligent Manufacture, andIndustrial Automation (ICAMIMIA), pp. 46–48 (2015)

23. Tian, X., Tian, Y.: The Elaboration of Ancient andModern FamousKai Calligraphy. Jiangxi Fine Arts Press, Jiangxi (2007)

24. Xiao, X., Jin, L., Yang, Y., Yang, W., Sun, J., Chang, T.: Buildingfast and compact convolutional neural networks for offline hand-written Chinese character recognition. Pattern Recognit. 72, 72–81(2017)

25. Yao, F., Shao, G., Yi, J.: Trajectory generation of the writing-brushfor a robot arm to inherit block-style Chinese character calligraphytechniques. Adv. Robot. 18(3), 331–356 (2004)

26. Yi, T., Lian, Z., Tang, Y., Xiao, J.: A data-driven personalized dig-ital ink for Chinese characters. In: Gurrin, C., Hopfgartner, F.,Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MultiMe-dia Modeling, pp. 254–265. Springer, New York (2014). https://doi.org/10.1007/978-3-319-04114-8

27. Zeng, H., Huang, Y., Chao, F., Zhou, C.: Survey of robotic callig-raphy research. CAAI Trans. Intell. Syst. 11(1), 15–26 (2016)

28. Zhang, Z.,Wu, J., Yu, K.: Chinese calligraphy specific style render-ing system. In: Proceedings of the ACM International Conferenceon Digital Libraries, pp. 99–108 (2010)

Publisher’s Note Springer Nature remains neutral with regard to juris-dictional claims in published maps and institutional affiliations.

123

Page 13: Intelligent Chinese calligraphy beautification from ...calligraphy and physically write it on paper is interesting because the physical form of calligraphy, with the scent of ink and

Intelligent Chinese calligraphy beautification from handwritten characters for robotic writing 1205

Xinyue Zhang is currently a mas-ter student of College of Infor-mation Engineering at NorthwestA&F University. She is engagedin research on computer graphicsand robotics. She received a B.E.from North China Electric PowerUniversity in 2016.

Yuanhao Li is currently an under-graduate student of College ofInformation Engineering at North-west A&F University. He isengaged in research on artificialintelligence and robotics.

Zhiyi Zhang is currently a pro-fessor of College of InformationEngineering at Northwest A&FUniversity. He is engaged inresearch on computer graphics andcomputer-aided design. Hereceived a B.E. in the Departmentof Chemistry from Shanxi Uni-versity, China and M.E. and D.E.in Computer Science from IwateUniversity, Japan, in 1998, 2004,and 2007, respectively. He joinedNorthwest A&F University as anAssociate Professor in 2007. Heworked as a visiting researcher at

Iwate University, Japan (2014–2015), supported by CSC.

Kouichi Konno is a professorof Faculty of Science and Engi-neering at Iwate University. Hereceived a B.S. in Information Sci-ence in 1985 from the Univer-sity of Tsukuba. He earned hisDr. Eng. in precision machineryengineering from the Universityof Tokyo in 1996. He joined thesolid modeling project at RICOHfrom 1985 to 1999 and the XVLproject at Lattice Technology in2000. He worked as an associateprofessor of Faculty of Engineer-ing at Iwate University from 2001

to 2009. His research interests include virtual reality, geometric mod-eling, 3D measurement systems, and computer graphics. He is a mem-ber of EuroGraphics.

Shaojun Hu is currently an asso-ciate professor of College of Infor-mation Engineering at NorthwestA&F University. He is engagedin research on Computer Graph-ics and Computer Animation. Hereceived a B.E. and M.E. at North-west A&F University, China, anda D.E. at Iwate University, Japan,in 2003, 2006, and 2009, respec-tively. He joined Northwest A&FUniversity as a Lecturer in 2010and became an Associate Profes-sor in 2014. He worked as a Post-Doc researcher at TU Wien

(2014–2015) funded by Eurasia Pacific Uninet scholarship and a vis-iting researcher at the University of Tokyo (2016–2017) supported byCSC. He is a member of CCF and ACM.

123