Features for handwriting recognition

| 2

The challenge

“Rappt JD 10 Feb no 175, om machtiging om af”

| 3

Short processing pipeline

“machtiging”

Feature extraction

Classification

82,34,66,…0.12

“machtiging”

Learning

| 4

Processing pipeline

Feature extraction

Classification

Preprocessing

| 5

Input image types

› Color:

› Grayscale:

› Binary:

Preprocessing

› Goal: enhance the foreground while reducing other visual symptoms (stains, noise, pictures, ...)

› Methods:• Contrast stretching• Highpass filtering• Despeckling• Change color representation (RGB, HSV,

grayscale, black/white, …)• Remove selected connected components ()• …

| 6

| 7

Connected components

| 8

Processing pipeline

Segmentation

Feature extraction

Classification

Preprocessing

| 9

Object of classification› Sentences› Words› Characters

(use grammar)(use dictionary)(use alphabet)

| 10

Object representations

› Image› Unordered vectors (in a coco)› Contour vectors› On-line vectors› Skeleton image› Skeleton vectors

(x, y)i

(x, y)k

(x, y)k

(x, y)k

I(x, y)

I(x, y)

| 11

A full processing pipeline

Segmentation

Normalization

Feature extraction

Classification

Preprocessing

| 12

Invariance

› Luminance / contrast› Position› Size› Rotation› Shear› Writer style› Ink thickness› …

| 13

Invariance by normalization

› Luminance / contrast› Position› Size› Rotation› Shear› Writer style› Ink thickness› …

Center on center of gravity

Contrast stretching

Scale to standard

size

| 14

Invariance by trying many deformations› Luminance / contrast› Position› Size› Rotation› Shear› Writer style› Ink thickness› …

Try different scale factors

Try different rotations

… and use the best recognition result

Try different deformation

s

| 15

Invariance by using invariant features

› Luminance / contrast› Position› Size› Rotation› Shear› Writer style› Ink thickness› …

Zernike invariant moments

| 16

A full processing pipeline

Segmentation

Normalization

Feature extraction

Classification

Preprocessing

82,34,66,…

| 17

Feature ROI types

› Whole object› Zones› Windowing

| 18

Whole object (“wholistic”)

| 19

Zones

| 20

Windowing

| 21

Feature types

› Image itself› Statistical› Structural› Abstract

› Image (off-line) features (1—20)› Contour / on-line features (21 – 28)

| 22

Feature 1 – 3

› Connected component images

› Scaled image

› Distance transform (on whiteboard)

| 23

Feature 4: density histogram

| 24

Feature 5: radon transform

| 25

Feature 6: run count pattern

3

6

2 3

| 26

Feature 7: run length pattern

avg

stdev

avg

stdev

| 27

Feature 8: Autocorrelation

| 28

Feature 9: Polar zones

| 29

Feature 10: radial zones (tip!)

| 30

Feature 11: zone histograms

Feature 12: Hinge

| 31

(By Marius Bulacu)

Feature 13: Fraglets

| 32

| 33

Regelmatigheden

Singulariteiten

Feature 14: J.C. Simon (1/2)

| 34

"million" ==> convex:concave:3(north:concave) :(north:LOOP):concave:(north:LOOP) :concave:north :concave:HOLE :2(convex:concave)

(J.-C. Simon, 1989)

Feature 14: J.C. Simon (2/2)

| 35

Feature 15: Structure of background (1/3)

| 36

Feature 15: Structure of background (2/3)

| 37

Feature 15: Structure of background (3/3)

| 38

Feature 16: Structure of foreground + background

| 39

Feature 17: Fourier transform (1/2)

From: http://ccp.uchicago.edu/~dcbradle/pages/5.23.06.html

| 40

Feature 17: Fourier transform (2/2)

Fig. 1 and 3 from: http://www.csse.uwa.edu.au/~wongt/matlab.html

Fig. 2 from: http://www.chemicool.com/definition/fourier_transform.html

| 41

Feature 18: Wavelet transform

From: http://www.regonaudio.com/Audio%20Measurement%20via%20Wavelets.html

| 42

Feature 19: Hu invariant moments

dxdyyxyxiM q

D

pqp ).,(,

0,0M area of the object

0,11,0 ,MM center of mass

Slide from: http://www.cedar.buffalo.edu/~govind/CSE717/lectures/CSE717_3.ppt

› Invariant for scale, position and rotation

› Derived from moments› Moments describe the image distribution with

respect to its axes › Works on (x, y) vectors

| 43

Feature 20: Zernike moments

From: Trier, O. D., Jain, A. K., and Taxt, T. (1996). Feature extraction methods for character recognition - a survey. Pattern Recognition,29:641–662.

| 44

Feature 21 – 28: Contour features

› (cos, sin) of running angle› (cos, sin) of running angular difference› Angular difference› Fourier transform› Ink density (horizontal or vertical)› Radon transform: (ink density, computed radially from

the c.o.g.)› Angular histogram› Curvature scale space ()

| 45

Feature 28: Curvature scale space

From: http://www.christine.oppe.info/blog/category/formen-und-farben/formenvergleich/

pos

itera

tion

End