Handwriting Recognition Handwriting Recognition

CPSC 4600 @ UTC/CSE

Handprint Recognition aims to design systems Handprint Recognition aims to design systems which are able to recognize handwriting of which are able to recognize handwriting of natural language natural language

Methods and recognition rates depend on the Methods and recognition rates depend on the level of constraints on handwriting.level of constraints on handwriting.

The constraints are mainly characterized by the:The constraints are mainly characterized by the:– types of handwritingtypes of handwriting– number of scriptorsnumber of scriptors– size of the vocabulary size of the vocabulary – spatial layout. spatial layout.

Handprint RecognitionHandprint Recognition

Methods and Strategies Methods and Strategies

Recognition strategies heavily depends on Recognition strategies heavily depends on the the nature of the datanature of the data to be recognized. to be recognized. In the In the cursivecursive case, the problem is made case, the problem is made complex by the fact that the writing is complex by the fact that the writing is fundamentally ambiguous as the letters in the fundamentally ambiguous as the letters in the word are generally linked together, poorly written word are generally linked together, poorly written and may even be missing.and may even be missing.On the contrary, hand On the contrary, hand printedprinted word recognition word recognition is more related to printed word recognition, the is more related to printed word recognition, the individual letters composing the word being individual letters composing the word being usually much easier to isolate and to identify. usually much easier to isolate and to identify.

Character Recognition techniques can be classified Character Recognition techniques can be classified according to two criteria: according to two criteria: – the way preprocessing is performed on the data the way preprocessing is performed on the data – the type of the decision algorithmthe type of the decision algorithm

Preprocessing techniques include :Preprocessing techniques include :– the use of global transforms (correlation, Fourier descriptors, the use of global transforms (correlation, Fourier descriptors,

etc.)etc.)– local comparison (local density, intersections with straight lines, local comparison (local density, intersections with straight lines,

variable masks, etc.) variable masks, etc.) – geometrical or topological characteristics (strokes, loops, geometrical or topological characteristics (strokes, loops,

openings, diacritical marks, skeleton, etc.)openings, diacritical marks, skeleton, etc.)

Decision methods include: Decision methods include: – various statistical methods, various statistical methods, – neural networks, structural matching (on trees, chains, etc.) neural networks, structural matching (on trees, chains, etc.) – stochastic processing (Markov chains, etc.). stochastic processing (Markov chains, etc.).

Character RecognitionCharacter Recognition

Two main types of strategies have been applied Two main types of strategies have been applied to this problem:to this problem:– the holistic approach - recognition is the holistic approach - recognition is globally globally

performed on the whole representation of wordsperformed on the whole representation of words and and there is no attempt to identify characters individually.there is no attempt to identify characters individually.

The main advantage of holistic methods is that they avoid The main advantage of holistic methods is that they avoid word segmentation word segmentation

– the analytical approach - deal with the analytical approach - deal with several levels of several levels of representationrepresentation corresponding to increasing levels of corresponding to increasing levels of abstraction (usually the feature level, the grapheme or abstraction (usually the feature level, the grapheme or pseudo-letter level and the word level). Words are not pseudo-letter level and the word level). Words are not considered as a whole, but as sequences of smaller considered as a whole, but as sequences of smaller size units which must be easily related to characters in size units which must be easily related to characters in order to make recognition independent from a specific order to make recognition independent from a specific vocabulary vocabulary

Word RecognitionWord Recognition

Form-based Handprint RecognitionForm-based Handprint Recognition

National Institute of Standards and National Institute of Standards and Technology (NIST) released to the public Technology (NIST) released to the public a standard reference form-based a standard reference form-based handprint recognition system for handprint recognition system for evaluating optical character recognition evaluating optical character recognition (OCR) in 1994.(OCR) in 1994.

http://www.utc.edu/Faculty/Li-Yang/CPSC415/4-Handwriting/hsfsys2.pdf

The NIST system is designed to read the hand The NIST system is designed to read the hand printed characters written on a printed characters written on a Handwriting Handwriting Sample Forms (HSF)Sample Forms (HSF). . The form is designed to collect a large sample to The form is designed to collect a large sample to handwriting to support handprint recognition handwriting to support handprint recognition research. research. NIST Special Database 19 (SD19) contains NIST Special Database 19 (SD19) contains 3669 completed forms, each filled by a unique 3669 completed forms, each filled by a unique writer, and scanned binary at 11.8 pixels per writer, and scanned binary at 11.8 pixels per millimeter. millimeter. The dataset also contains over 800,000 The dataset also contains over 800,000 segmented and labeled characters images from segmented and labeled characters images from these forms.these forms.

Form-based Handprint RecognitionForm-based Handprint Recognition

There is a blank There is a blank form provided form provided that can be that can be printed, filled in, printed, filled in, scanned and scanned and recognized.recognized.

System ComponentsSystem Components

Batch InitializationBatch InitializationLoad Form ImageLoad Form ImageRegister Form ImageRegister Form ImageRemove From BoxRemove From BoxIsolate Lines of HandprintIsolate Lines of HandprintSegment Text LinesSegment Text LinesNormalize CharactersNormalize CharactersExtract Feature VectorsExtract Feature VectorsClassify CharactersClassify CharactersSpell-correct Text LinesSpell-correct Text Lines

Batch InitializationBatch Initialization

Load pre-computed items from trainingLoad pre-computed items from training– A list of images files to be processedA list of images files to be processed– Coordinate locations of dominant form structures Coordinate locations of dominant form structures

used for form registrationused for form registration– Spatial template containing the coordinate locationSpatial template containing the coordinate location– Basis functions used for feature extractionBasis functions used for feature extraction– Neural network weights for classificationNeural network weights for classification– Dictionaries for spelling correctionDictionaries for spelling correction

Four types of fields: numeric, lowercase, Four types of fields: numeric, lowercase, uppercase, and preamble paragraphuppercase, and preamble paragraphEach type of fields requires a separate set of Each type of fields requires a separate set of basis functions and neural network weights. basis functions and neural network weights.

Register From ImageRegister From Image

To reliably isolate the handprint on a formTo reliably isolate the handprint on a formForm registration automatically estimates the Form registration automatically estimates the amount of rotation and translation in the image. amount of rotation and translation in the image. Because most forms contains a fixed Because most forms contains a fixed configuration of vertical and horizontal lines, we configuration of vertical and horizontal lines, we trace parallel ray across the image accumulating trace parallel ray across the image accumulating the number of black pixels along each ray. the number of black pixels along each ray. A range of ray angles are sample, the angles A range of ray angles are sample, the angles producing the maximum response is used to producing the maximum response is used to estimate the rotational skew. estimate the rotational skew.

A prototypical from is A prototypical from is scanned, scanned, its rotational distortion is its rotational distortion is automatically measured automatically measured and removed, and and removed, and the position of the the position of the detected dominant line detected dominant line s are stored for future s are stored for future registration. registration. The image is the result The image is the result of logically ORing of logically ORing corresponding pixels corresponding pixels across a set of 500 across a set of 500 registered images. registered images.

Remove Form BoxRemove Form Box

Given a field sub-image, Given a field sub-image, black pixels corresponding to black pixels corresponding to the handwritingthe handwriting must be separated from must be separated from the black pixels the black pixels corresponding to the formcorresponding to the form. . We need locate the box within the field sub-image, and We need locate the box within the field sub-image, and intelligently removes the sides so as to preserve intelligently removes the sides so as to preserve overlapping characters. overlapping characters. The sides of the box are detected using a run-based The sides of the box are detected using a run-based techniques that tracks the longest runs across the sub-techniques that tracks the longest runs across the sub-image. image. Overlapping character stokes are identified using spatial Overlapping character stokes are identified using spatial cures, and only pixels that are distinctly part of the form’s cures, and only pixels that are distinctly part of the form’s box are removed. box are removed.

Remove Form BoxRemove Form Box

Isolate Lines of HandprintIsolate Lines of Handprint

A A connected component is defined as the largest set of black pixels where each pixel is a direct neighbor of at least one other black pixel in the component.For multiple-line responses There are no lines provided within this paragraph box to guide a writer. Bottom-up approach to isolate the lines of handprint within a paragraph.Each component is represented by its geometric center. To reconstruct the handprinted lines of text, a nearest neighbor search is performed left-to-right and top-to-bottom through the system of 2-dimensional points.

Isolate Lines of HandprintIsolate Lines of Handprint

Segment Text LinesSegment Text Lines

Connected components are used as first-order approximations to single and complete characters. Connected components frequently represent single characters and are computed very quickly.Errors occur when characters touch one another and when characters are written with disconnected strokes (naturally occurring with dotted letters).

A simple adaptive model of writing style

In a simple adaptive model of writing style, fragmented characters are reconstructed, multiple characters are split, and noise components are identified and discarded.M. D. Garris, “Component-Based Handprint Segmentation Using Adaptive Writing Style Model,” NIST Internal Report 5843, June 1996.

Model Writing StyleModel Writing Style

ToTo adapt to variations in handwriting style, one needs to be able to statistically capture how much black ink (or pixels) in an image is likely to constitute a single character.Two simple statistical features are measured from each isolated image of handwriting– The estimated stroke width (esw) approximates the width of the

lines comprising the characters.– The estimate character height (ech) is to find the maximum

height of all the connected components in the image. – Standard stroke pixel (ssp) = square of one stroke width– Standard stroke area (ssa)Standard stroke area (ssa) = = estimated stroke width * estimate

character height

If (component.area < (0.5 * ssa) then Noisewhere structure member (a) is the pixel area of the component (c) and ssa is the pixel area of a standard stroke width.If (component.width < (2 * esw)) && (component.height < (3 * esw)) then Dot

where structure member (w) is the pixel width of the component (c)

Characters that required the Characters that required the merging of connected componentsmerging of connected components

Multiple Character DetectionMultiple Character Detection

Before one can split touching characters, one must be able to detect that multiple characters exist in a component image.a simple aspect ratio (ar) was tested.

where w is the width of the component, and ech is the estimated character height for the field.

The larger the width is to the height, the more likely the component contains multiple characters.

A training set of single and touching character components was used to compute a range of aspect ratio samples, and a threshold was empirically derived.

Multiple Character DetectionMultiple Character Detection

standard stroke count (ssc) or ssc = p/ssa where p is the black pixel count of the component.

Vertically Straight CutVertically Straight Cut

An example of multiple characters

A component determined to contain multiple touching characters must be further analyzed to derive a strategy for splitting the characters.

Vertically Straight CutVertically Straight CutPerpendicular distances are computed from the left and right feature points to the detector line and the larger of the two distances is stored along with the x-position of the vertical cut.

By minimizing the maximum perpendicular distances across the range of cuts, the vertical cut is selected whose left and right pieces both contain maximal pixel data and both pieces qualify as single characters.

Contoured Cut PathContoured Cut PathA single straight cut does not satisfactorily divide the component. In these cases, a more sophisticated non-straight path is required.Starting at the x-position of the optimal vertical cut, a search (or trace) is initiated from the top of the component downwards and from the bottom of the component upwards.The trace downwards (the top-trace) performs much like sand being dribbled down the side of a complex surface.

Segment Text LinesSegment Text Lines

Normalize Characters and Extract Normalize Characters and Extract Feature VectorsFeature Vectors

The segmented character images vary greatly in size, slant, and shape. Image normalization is performed to deal with the size and slant of writing, leaving the recognition process primarily the task of differentiating characters by variation in shape.The Karhunen Loève (KL) transform is applied to these binary pixel vectors in order to reduce dimensionality, suppress noise, and produce optimally compact features (in terms of variance) for classification.

Classify Characters

Once segmented characters are represented as feature vectors, a whole host of different pattern classification techniques can be applied.

Probabilistic Neural Network (PNN), or

Multi-Layer Perceptron (MLP) neural network

Spell-Correct Text Line(s)

segmented character images have been extracted from the handprinted paragraph, sorted into reading order line by line, and classified.

This results in one long contiguous character stream for each line in the paragraph.

Words are parsed from each line of raw classifications by applying the preloaded dictionary.

Spell-Correct Text Line(s)

Spell-Correct Text Line(s)

Lab ExercisesLab Exercises

Download and Install NIST form-based Download and Install NIST form-based handprint software.  Test and document handprint software.  Test and document the process of handwriting recognition.  the process of handwriting recognition.  http://www.itl.nist.gov/iaui/vip/databases/defs/nist_ocr.html   

ReferencesReferencesM. D. Garris and P. J. Grother, “Generalized Form Registration Using Structure-Based Techniques,” NIST Internal Report 5726 and in Proceedings of the Fifth Annual Symposium on Document Analysis and Information Retrieval, pp. 321-334, UNLV, April 1996.M. D. Garris, “Method and Evaluation of Character Stroke Preservation of Handprint Recognition,” NIST Internal Report 5687, July 1995, and in Proceedings of Document Recognition III, Vol. 2660, pp. 321-332, SPIE, San Jose, February 1996.M. D. Garris, “Component-Based Handprint Segmentation Using Adaptive Writing Style Model,” NIST Internal Report 5843, June 1996.W. Postl, “Method for Automatic Correction of Character Skew in the Acquisition of a Text Original in the Form of Digital Scan Results,” United States Patent Number 4,723,297, February 1988.

GraphologyGraphology

Graphology or Handwriting Analysis is a science Graphology or Handwriting Analysis is a science of interpreting a person's character from his/her of interpreting a person's character from his/her personal handwriting. personal handwriting.

Handwriting analyzing can tell a lot about Handwriting analyzing can tell a lot about personality personality

Large companies use graphology (handwriting analyzing) to check job applications.

The police still use handwriting experts to determine who wrote what.

GraphologyGraphology

Coupled with psychology and knowledge of human Coupled with psychology and knowledge of human behavior, it can be used for recruitment, marriage behavior, it can be used for recruitment, marriage compatibility, career guidance, and child compatibility, career guidance, and child development, etc. development, etc. Signature is easy to collect, but not the best way to Signature is easy to collect, but not the best way to analyze character from handwriting , for several analyze character from handwriting , for several reasons.reasons.– Firstly, the Signature is sometimes illegible and different Firstly, the Signature is sometimes illegible and different

from the normal handwriting.from the normal handwriting.– Secondly, the lone signature does not give enough Secondly, the lone signature does not give enough

words and letters to help the graphologist in his/her words and letters to help the graphologist in his/her judgment. judgment.

GraphologyGraphology

Graphology can analyze a person's personality without Graphology can analyze a person's personality without the person’s knowledge that he/she is being analyzed. the person’s knowledge that he/she is being analyzed. Distortion of the result may be cause by the subject's Distortion of the result may be cause by the subject's knowledge that he/she is being analyzed. This is knowledge that he/she is being analyzed. This is common in most Question-&-Answer type of personality common in most Question-&-Answer type of personality analysis tools. analysis tools. This behavior analysis is more accurate than putting the This behavior analysis is more accurate than putting the subject under unnatural stress of a long questionnaire. subject under unnatural stress of a long questionnaire. A single personality questionnaire may reveal a single A single personality questionnaire may reveal a single dimension of personality. But two handwriting samples of dimension of personality. But two handwriting samples of the same person at different occasion may reveals the same person at different occasion may reveals different behavior characteristics. different behavior characteristics.

GraphologyGraphology

What is Graphoanalysis?What is Graphoanalysis?

Graphoanalysis is a scientific system of Graphoanalysis is a scientific system of identifying and assessing the character and identifying and assessing the character and personality of an individual through a study of personality of an individual through a study of handwriting. The techniques used are based on handwriting. The techniques used are based on a well-defined, standardized method of:a well-defined, standardized method of:– identifying strokes, identifying strokes, – relating these strokes to specific personality traits, relating these strokes to specific personality traits,

and and – evaluating the relative strength of the interrelated evaluating the relative strength of the interrelated

traits. traits.

The Hidden Meaning of HandwritingThe Hidden Meaning of Handwriting

Handwriting gives us access to inside secrets Handwriting gives us access to inside secrets about the hidden meaning of handwritingabout the hidden meaning of handwriting

If you received a note like the one belowIf you received a note like the one below– What would you be able to tell about the writer? The What would you be able to tell about the writer? The

words are friendly enough!words are friendly enough!– If you look a little closer you will see that there are a If you look a little closer you will see that there are a

number of conflicting signs. number of conflicting signs. – Can you see the danger signals?  There are red flags Can you see the danger signals?  There are red flags

popping up all over the placepopping up all over the place

An understanding of handwriting analysis will An understanding of handwriting analysis will quickly show you that this writer is dogmatic and quickly show you that this writer is dogmatic and dictatorial, emotionally unstable, bad tempered dictatorial, emotionally unstable, bad tempered and possibly even violent! and possibly even violent! It's even possible to find the underlying reasons It's even possible to find the underlying reasons for the dangerous signals and to understand for the dangerous signals and to understand why the writer has so much personal conflict in why the writer has so much personal conflict in his life.his life.You'll be surprised to discover how much detail You'll be surprised to discover how much detail can be extracted from just a single page of can be extracted from just a single page of handwriting. handwriting. Of course, to be able to read these signs Of course, to be able to read these signs yourself, you will need to have some yourself, you will need to have some understanding of handwriting analysis first. understanding of handwriting analysis first.

GraphologyGraphology

Handwriting GuidelinesHandwriting Guidelines

Get a sheet of paper, sign your name, and start Get a sheet of paper, sign your name, and start to analyze with these guidelines:to analyze with these guidelines:– Writing that leans to the right shows good Writing that leans to the right shows good

communication skills. communication skills. – Straight, vertical writing shows independence and Straight, vertical writing shows independence and

stability. stability. – Writing that leans to the left may mean you find it Writing that leans to the left may mean you find it

difficult to communicate. difficult to communicate. – Small writing can be a sign of modesty. Small writing can be a sign of modesty. – Large writing indicates enthusiasm and generosity.Large writing indicates enthusiasm and generosity.

Exercise ( 15 minutes)Exercise ( 15 minutes)

Get a paper and write the following Get a paper and write the following sentences:sentences:– Go west young man and grow up with the Go west young man and grow up with the

countrycountry– It is Valentino’s day today, so I will give her …It is Valentino’s day today, so I will give her …

Give it to your group member to analyze it Give it to your group member to analyze it – based on the guidelines.– based on the guidelines.

Lab ExerciseLab Exercise

Go to the following site and do an online Go to the following site and do an online handwriting analysis.handwriting analysis.– http://www.myhandwriting.com/index.html– http://www.compatamate.com/HWAReport/ind

ex.html